[ClusterLabs] Resources wont start on new node unless it is the only active node
Ryan Anstey
ryan at treasuremart.net
Tue Nov 8 16:07:37 EST 2016
Actually I had those two off just to simplify since it was the RBD failing
that caused the rest to fail afterwards. I seem to have fixed the problem
though. I used "crm move grp_unifi h4 force" and it started it up fine. I
don't know why the force option made it work, but everything appears to be
working without any issues now.
On Tue, Nov 8, 2016 at 12:25 PM Andreas Kurz <andreas.kurz at gmail.com> wrote:
> Hi,
>
> On Tue, Nov 8, 2016 at 7:54 PM, Ryan Anstey <ryan at treasuremart.net> wrote:
>
> I've been running a ceph cluster with pacemaker for a few months now.
> Everything has been working normally, but when I added a fourth node it
> won't work like the others, even though their OS is the same and the
> configs are all synced via salt. I also don't understand pacemaker that
> well since I followed a guide for it. If anyone could steer me in the right
> direction I would greatly appreciate it. Thank you!
>
> - My resources only start if the new node is the only active node.
> - Once started on new node, if they are moved back to one of the original
> nodes, it won't go back to the new one.
> - My resources work 100% if I start them manually (without pacemaker).
> - (In the logs/configs below, my resources are named "unifi", "rbd_unifi"
> being the main one that's not working.)
>
>
> looking at your configuration below I would start with setting these
> resources to be "managed":
>
> fs_unifi
> lxc_unifi
>
> and to also start that resources.
>
> crm resource manage ...
> crm resource start ...
>
> Regards,
> Andreas
>
>
>
>
> Log when running cleaning up the resource on the NEW node:
>
> Nov 08 09:25:20 h4 Filesystem(fs_unifi)[18044]: WARNING: Couldn't find
> device [/dev/rbd/rbd/unifi]. Expected /dev/??? to exist
> Nov 08 09:25:20 h4 lrmd[3564]: notice: lxc_unifi_monitor_0:18018:stderr [
> unifi doesn't exist ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation lxc_unifi_monitor_0: not
> running (node=h4, call=484, rc=7, cib-update=390, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: h4-lxc_unifi_monitor_0:484 [ unifi
> doesn't exist\n ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation fs_unifi_monitor_0: not
> running (node=h4, call=480, rc=7, cib-update=391, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation rbd_unifi_monitor_0: not
> running (node=h4, call=476, rc=7, cib-update=392, confirmed=true)
>
> Log when running cleaning up the resource on the OLD node:
>
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: notice: State transition S_IDLE ->
> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 cib[11389]: warning: A-Sync reply to crmd failed: No
> message of desired type
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:22 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:25 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:25 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start rbd_unifi (h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start fs_unifi (h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start lxc_unifi (h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Calculated Transition 119:
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Processing graph 119
> (ref=pe_calc-dc-1478625686-648) derived from
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 12: monitor
> rbd_unifi_monitor_0 on h4
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 9: monitor
> rbd_unifi_monitor_0 on h3 (local)
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 6: monitor
> rbd_unifi_monitor_0 on h2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 3: monitor
> rbd_unifi_monitor_0 on h1
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 13: monitor
> fs_unifi_monitor_0 on h4
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 10: monitor
> fs_unifi_monitor_0 on h3 (local)
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 7: monitor
> fs_unifi_monitor_0 on h2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 4: monitor
> fs_unifi_monitor_0 on h1
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 14: monitor
> lxc_unifi_monitor_0 on h4
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 11: monitor
> lxc_unifi_monitor_0 on h3 (local)
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 8: monitor
> lxc_unifi_monitor_0 on h2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 5: monitor
> lxc_unifi_monitor_0 on h1
> Nov 08 09:21:26 h3 crmd[11394]: notice: Operation fs_unifi_monitor_0: ok
> (node=h3, call=446, rc=0, cib-update=767, confirmed=true)
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 10 (fs_unifi_monitor_0) on
> h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: notice: Transition aborted by
> fs_unifi_monitor_0 'create' on h3: Event failed
> (magic=0:0;10:119:7:2bc5b523-b947-4035-b296-4c099646298b, cib=0.1445.8,
> source=match_graph_event:381, 0)
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 10 (fs_unifi_monitor_0) on
> h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: notice: Operation lxc_unifi_monitor_0: ok
> (node=h3, call=450, rc=0, cib-update=768, confirmed=true)
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 11 (lxc_unifi_monitor_0)
> on h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 11 (lxc_unifi_monitor_0)
> on h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: notice: Operation rbd_unifi_monitor_0: ok
> (node=h3, call=442, rc=0, cib-update=769, confirmed=true)
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 9 (rbd_unifi_monitor_0) on
> h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: warning: Action 9 (rbd_unifi_monitor_0) on
> h3 failed (target: 7 vs. rc: 0): Error
> Nov 08 09:21:26 h3 crmd[11394]: notice: Transition 119 (Complete=13,
> Pending=0, Fired=0, Skipped=0, Incomplete=4,
> Source=/var/lib/pacemaker/pengine/pe-input-739.bz2): Complete
> Nov 08 09:21:26 h3 pengine[11393]: notice: Calculated Transition 120:
> /var/lib/pacemaker/pengine/pe-input-740.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Processing graph 120
> (ref=pe_calc-dc-1478625686-661) derived from
> /var/lib/pacemaker/pengine/pe-input-740.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Transition 120 (Complete=0,
> Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-740.bz2): Complete
> Nov 08 09:21:26 h3 crmd[11394]: notice: State transition
> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
> origin=notify_crmd ]
>
> Output of crm configure show:
>
> node 167838209: h1 \
> attributes standby=off
> node 167838210: h2 \
> attributes standby=off
> node 167838211: h3 \
> attributes standby=off
> node 167838212: h4 \
> attributes standby=off
> rsc_template fs Filesystem \
> op start timeout=60s interval=0 \
> op stop timeout=60s interval=0
> rsc_template kvm VirtualDomain \
> params hypervisor="qemu:///system" \
> meta allow-migrate=true \
> op start timeout=120s interval=0 \
> op stop timeout=120s interval=0 \
> op monitor timeout=30 interval=10 depth=0
> rsc_template lxc lxc \
> params use_screen=true \
> op stop timeout=300s interval=0
> primitive fs_first @fs \
> params device="/dev/rbd/rbd/first" fstype=xfs
> directory="/srv/first" \
> meta target-role=Stopped
> primitive fs_fs1 @fs \
> params device="/dev/rbd/rbd/fs1" fstype=xfs
> directory="/var/lib/lxc/fs1"
> primitive fs_ld1 @fs \
> params device="/dev/rbd/rbd/ld1" fstype=xfs
> directory="/var/lib/lxc/ld1"
> primitive fs_ld2 @fs \
> params device="/dev/rbd/rbd/ld2" fstype=xfs
> directory="/var/lib/lxc/ld2"
> primitive fs_ld3 @fs \
> params device="/dev/rbd/rbd/ld3" fstype=xfs
> directory="/var/lib/lxc/ld3"
> primitive fs_ls1 @fs \
> params device="/dev/rbd/rbd/ls1" fstype=xfs
> directory="/var/lib/lxc/ls1"
> primitive fs_ls2 @fs \
> params device="/dev/rbd/rbd/ls2" fstype=xfs
> directory="/var/lib/lxc/ls2"
> primitive fs_ls3 @fs \
> params device="/dev/rbd/rbd/ls3" fstype=xfs
> directory="/var/lib/lxc/ls3"
> primitive fs_ls4 @fs \
> params device="/dev/rbd/rbd/ls4" fstype=xfs
> directory="/var/lib/lxc/ls4"
> primitive fs_ls5 @fs \
> params device="/dev/rbd/rbd/ls5" fstype=xfs
> directory="/var/lib/lxc/ls5"
> primitive fs_rdosrv @fs \
> params device="/dev/rbd/rbd/rdosrv" fstype=xfs
> directory="/var/lib/lxc/rdosrv"
> primitive fs_unifi @fs \
> params device="/dev/rbd/rbd/unifi" fstype=xfs
> directory="/var/lib/lxc/unifi" \
> meta target-role=Stopped is-managed=false
> primitive kvm_foxy @kvm \
> params config="/srv/cluster/containers/foxy.xml" \
> utilization cpu=8 hv_memory=4000 \
> meta target-role=Stopped is-managed=true
> primitive kvm_ws5 @kvm \
> params config="/srv/cluster/containers/ws5.xml" \
> utilization cpu=4 hv_memory=4096 \
> meta target-role=Started
> primitive lxc_fs1 @lxc \
> params container=fs1 config="/var/lib/lxc/fs1/config"
> primitive lxc_ld1 @lxc \
> params container=ld1 config="/var/lib/lxc/ld1/config" \
> meta target-role=Stopped
> primitive lxc_ld2 @lxc \
> params container=ld2 config="/var/lib/lxc/ld2/config"
> primitive lxc_ld3 @lxc \
> params container=ld3 config="/var/lib/lxc/ld3/config"
> primitive lxc_ls1 @lxc \
> params container=ls1 config="/var/lib/lxc/ls1/config"
> primitive lxc_ls2 @lxc \
> params container=ls2 config="/var/lib/lxc/ls2/config"
> primitive lxc_ls3 @lxc \
> params container=ls3 config="/var/lib/lxc/ls3/config"
> primitive lxc_ls4 @lxc \
> params container=ls4 config="/var/lib/lxc/ls4/config"
> primitive lxc_ls5 @lxc \
> params container=ls5 config="/var/lib/lxc/ls5/config" \
> meta target-role=Started
> primitive lxc_rdosrv @lxc \
> params container=rdosrv config="/var/lib/lxc/rdosrv/config"
> primitive lxc_unifi @lxc \
> params container=unifi config="/var/lib/lxc/unifi/config" \
> meta target-role=Stopped is-managed=false
> primitive rbd_first ocf:ceph:rbd \
> params name=first \
> meta target-role=Stopped
> primitive rbd_fs1 ocf:ceph:rbd \
> params name=fs1
> primitive rbd_ld1 ocf:ceph:rbd \
> params name=ld1
> primitive rbd_ld2 ocf:ceph:rbd \
> params name=ld2
> primitive rbd_ld3 ocf:ceph:rbd \
> params name=ld3
> primitive rbd_ls1 ocf:ceph:rbd \
> params name=ls1
> primitive rbd_ls2 ocf:ceph:rbd \
> params name=ls2
> primitive rbd_ls3 ocf:ceph:rbd \
> params name=ls3
> primitive rbd_ls4 ocf:ceph:rbd \
> params name=ls4
> primitive rbd_ls5 ocf:ceph:rbd \
> params name=ls5
> primitive rbd_rdosrv ocf:ceph:rbd \
> params name=rdosrv \
> meta target-role=Started
> primitive rbd_unifi ocf:ceph:rbd \
> params name=unifi \
> meta target-role=Started
> group grp_first rbd_first fs_first \
> meta is-managed=true
> group grp_fs1 rbd_fs1 fs_fs1 lxc_fs1 \
> meta is-managed=true target-role=Started
> group grp_ld1 rbd_ld1 fs_ld1 lxc_ld1 \
> meta target-role=Stopped
> group grp_ld2 rbd_ld2 fs_ld2 lxc_ld2 \
> meta target-role=Stopped
> group grp_ld3 rbd_ld3 fs_ld3 lxc_ld3 \
> meta target-role=Started
> group grp_ls1 rbd_ls1 fs_ls1 lxc_ls1 \
> meta target-role=Started
> group grp_ls2 rbd_ls2 fs_ls2 lxc_ls2 \
> meta is-managed=true
> group grp_ls3 rbd_ls3 fs_ls3 lxc_ls3 \
> meta target-role=Started
> group grp_ls4 rbd_ls4 fs_ls4 lxc_ls4 \
> meta target-role=Stopped
> group grp_ls5 rbd_ls5 fs_ls5 lxc_ls5 \
> meta target-role=Started
> group grp_rdosrv rbd_rdosrv fs_rdosrv lxc_rdosrv \
> meta target-role=Started is-managed=true
> group grp_unifi rbd_unifi fs_unifi lxc_unifi \
> meta target-role=Started
> location cli-prefer-fs_ls5 fs_ls5 role=Started inf: h3
> location cli-prefer-grp_fs1 grp_fs1 role=Started inf: h4
> location cli-prefer-grp_ld1 grp_ld1 role=Started inf: h3
> location cli-prefer-grp_ld2 grp_ld2 role=Started inf: h2
> location cli-prefer-grp_ld3 grp_ld3 role=Started inf: h4
> location cli-prefer-grp_ls1 grp_ls1 role=Started inf: h2
> location cli-prefer-grp_ls2 grp_ls2 role=Started inf: h2
> location cli-prefer-grp_ls3 grp_ls3 role=Started inf: h4
> location cli-prefer-grp_ls4 grp_ls4 role=Started inf: h3
> location cli-prefer-grp_ls5 grp_ls5 role=Started inf: h4
> location cli-prefer-grp_rdosrv grp_rdosrv role=Started inf: h4
> location cli-prefer-grp_unifi grp_unifi role=Started inf: h4
> location cli-prefer-kvm_foxy kvm_foxy role=Started inf: h1
> location cli-prefer-kvm_ws5 kvm_ws5 role=Started inf: h1
> location cli-prefer-lxc_fs1 lxc_fs1 role=Started inf: h3
> location cli-prefer-lxc_ld3 lxc_ld3 role=Started inf: h3
> location cli-prefer-lxc_ls1 lxc_ls1 role=Started inf: h2
> location cli-prefer-lxc_ls3 lxc_ls3 role=Started inf: h2
> location cli-prefer-lxc_ls4 lxc_ls4 role=Started inf: h3
> location cli-prefer-lxc_ls5 lxc_ls5 role=Started inf: h3
> location cli-prefer-lxc_rdosrv lxc_rdosrv role=Started inf: h3
> location cli-prefer-lxc_unifi lxc_unifi role=Started inf: h3
> location cli-prefer-rbd_ls5 rbd_ls5 role=Started inf: h3
> location cli-prefer-rbd_rdosrv rbd_rdosrv role=Started inf: h4
> property cib-bootstrap-options: \
> have-watchdog=false \
> dc-version=1.1.13-20.1-6f22ad7 \
> cluster-infrastructure=corosync \
> cluster-name=hosteria \
> stonith-enabled=false \
> last-lrm-refresh=1478629707
> rsc_defaults rsc-options: \
> resource-stickiness=100
>
> Pacemaker logs:
>
> Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='rbd_unifi'] to master
> (origin=local/crmd/393)
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1445.14 2
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1445.15 0baf07ee77ccf8d61824f54c68f116b7
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=15
> Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h1/crmd/345, version=0.1445.14)
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1445.14 2
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1445.15 (null)
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=15
> Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h4/crmd/393, version=0.1445.14)
> Nov 08 09:35:00 [3567] h4 crmd: info: delete_resource: Removing
> resource rbd_unifi for d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 (root) on h4
> Nov 08 09:35:00 [3567] h4 crmd: info: notify_deleted: Notifying
> d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 on h4 that rbd_unifi was deleted
> Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='rbd_unifi'] to master
> (origin=local/crmd/394)
> Nov 08 09:35:00 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_modify operation for section crm_config to master
> (origin=local/crmd/396)
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1445.14 2
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1445.15 (null)
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:00 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=15
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h4/crmd/394, version=0.1445.15)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1445.15 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1446.0 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @epoch=1446, @num_updates=0
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: +
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
> @value=1478626500
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h4/crmd/396,
> version=0.1446.0)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='fs_unifi'] to master
> (origin=local/crmd/397)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1446.0 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1446.1 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=1
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h4/crmd/397, version=0.1446.0)
> Nov 08 09:35:01 [3567] h4 crmd: info: delete_resource: Removing
> resource fs_unifi for d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 (root) on h4
> Nov 08 09:35:01 [3567] h4 crmd: info: notify_deleted: Notifying
> d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 on h4 that fs_unifi was deleted
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1446.0 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1446.1 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=1
> Nov 08 09:35:01 [3562] h4 cib: info: cib_file_backup: Archived
> previous version as /var/lib/pacemaker/cib/cib-64.raw
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h1/crmd/346, version=0.1446.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1446.1 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.0 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @epoch=1447, @num_updates=0
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: +
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
> @value=1478626267
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h1/crmd/348,
> version=0.1447.0)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.0 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.1 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=1
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h1/crmd/349, version=0.1447.0)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.0 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.1 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=1
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h1/crmd/350, version=0.1447.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_file_write_with_digest: Wrote
> version 0.1446.0 of the CIB to disk (digest:
> 13247e2b093b5d2dee902604bc9a3208)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h1/crmd/352,
> version=0.1447.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_file_write_with_digest: Reading
> cluster configuration file /var/lib/pacemaker/cib/cib.111AJI (digest:
> /var/lib/pacemaker/cib/cib.Zv2YLg)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.1 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.2 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h1/crmd/353, version=0.1447.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.1 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.2 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h1']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h1/crmd/354, version=0.1447.2)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h1/crmd/356,
> version=0.1447.2)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.2 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.3 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=3
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h2/crmd/346, version=0.1447.2)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='fs_unifi'] to master
> (origin=local/crmd/398)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.2 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.3 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=3
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h4/crmd/398, version=0.1447.3)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.3 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.4 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=4
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h2/crmd/347, version=0.1447.4)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h2/crmd/349,
> version=0.1447.4)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.4 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.5 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=5
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h2/crmd/350, version=0.1447.4)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.4 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.5 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=5
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h2/crmd/351, version=0.1447.5)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h2/crmd/353,
> version=0.1447.5)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.5 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.6 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=6
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h2/crmd/354, version=0.1447.5)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.5 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1447.6 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=6
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h2']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h2/crmd/355, version=0.1447.6)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h2/crmd/357,
> version=0.1447.6)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_modify operation for section crm_config to master
> (origin=local/crmd/400)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='lxc_unifi'] to master
> (origin=local/crmd/401)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1447.6 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1448.0 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @epoch=1448, @num_updates=0
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: +
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
> @value=1478626501
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h4/crmd/400,
> version=0.1448.0)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1448.0 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1448.1 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=1
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h4/crmd/401, version=0.1448.0)
> Nov 08 09:35:01 [3567] h4 crmd: info: delete_resource: Removing
> resource lxc_unifi for d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 (root) on h4
> Nov 08 09:35:01 [3567] h4 crmd: info: notify_deleted: Notifying
> d44ab7d6-13f8-4a8c-a9f9-ffeeda6727a2 on h4 that lxc_unifi was deleted
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='lxc_unifi'] to master
> (origin=local/crmd/402)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_modify operation for section crm_config to master
> (origin=local/crmd/404)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1448.0 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1448.1 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=1
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h4']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h4/crmd/402, version=0.1448.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h4/crmd/404,
> version=0.1448.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1448.1 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1448.2 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h3/crmd/771, version=0.1448.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1448.1 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1448.2 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='rbd_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='rbd_unifi']: OK (rc=0,
> origin=h3/crmd/772, version=0.1448.2)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1448.2 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.0 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @epoch=1449, @num_updates=0
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: +
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']:
> @value=1478626267
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h3/crmd/774,
> version=0.1449.0)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_file_backup: Archived
> previous version as /var/lib/pacemaker/cib/cib-65.raw
> Nov 08 09:35:01 [3562] h4 cib: info: cib_file_write_with_digest: Wrote
> version 0.1449.0 of the CIB to disk (digest:
> f6be1cc446b06cf40d434f709a1e660e)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_file_write_with_digest: Reading
> cluster configuration file /var/lib/pacemaker/cib/cib.dq077Z (digest:
> /var/lib/pacemaker/cib/cib.HTyLSy)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.0 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.1 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=1
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h3/crmd/789, version=0.1449.0)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.0 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.1 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='fs_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=1
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='fs_unifi']: OK (rc=0,
> origin=h3/crmd/790, version=0.1449.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h3/crmd/792,
> version=0.1449.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.1 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.2 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h3/crmd/796, version=0.1449.1)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.1 2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.2 (null)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: --
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources/lrm_resource[@id='lxc_unifi']
> Nov 08 09:35:01 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=2
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_delete operation for section
> //node_state[@uname='h3']//lrm_resource[@id='lxc_unifi']: OK (rc=0,
> origin=h3/crmd/797, version=0.1449.2)
> Nov 08 09:35:01 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section crm_config: OK (rc=0, origin=h3/crmd/799,
> version=0.1449.2)
> Nov 08 09:35:05 [3564] h4 lrmd: info: process_lrmd_get_rsc_info: Resource
> 'rbd_unifi' not found (34 active resources)
> Nov 08 09:35:05 [3564] h4 lrmd: info: process_lrmd_rsc_register: Added
> 'rbd_unifi' to the rsc list (35 active resources)
> Nov 08 09:35:05 [3567] h4 crmd: info: do_lrm_rsc_op: Performing
> key=12:121:7:2bc5b523-b947-4035-b296-4c099646298b op=rbd_unifi_monitor_0
> Nov 08 09:35:05 [3564] h4 lrmd: info: process_lrmd_get_rsc_info: Resource
> 'fs_unifi' not found (35 active resources)
> Nov 08 09:35:05 [3564] h4 lrmd: info: process_lrmd_rsc_register: Added
> 'fs_unifi' to the rsc list (36 active resources)
> Nov 08 09:35:05 [3567] h4 crmd: info: do_lrm_rsc_op: Performing
> key=13:121:7:2bc5b523-b947-4035-b296-4c099646298b op=fs_unifi_monitor_0
> Nov 08 09:35:05 [3564] h4 lrmd: info: process_lrmd_get_rsc_info: Resource
> 'lxc_unifi' not found (36 active resources)
> Nov 08 09:35:05 [3564] h4 lrmd: info: process_lrmd_rsc_register: Added
> 'lxc_unifi' to the rsc list (37 active resources)
> Nov 08 09:35:05 [3567] h4 crmd: info: do_lrm_rsc_op: Performing
> key=14:121:7:2bc5b523-b947-4035-b296-4c099646298b op=lxc_unifi_monitor_0
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.2 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.3 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=3
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources:
> <lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="3:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;3:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h1" call-id="441" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h1/crmd/357,
> version=0.1449.3)
> Nov 08 09:35:05 [3564] h4 lrmd: notice: operation_finished: lxc_unifi_monitor_0:23285:stderr
> [ unifi doesn't exist ]
> Nov 08 09:35:05 [3567] h4 crmd: notice: process_lrm_event: Operation
> lxc_unifi_monitor_0: not running (node=h4, call=499, rc=7, cib-update=405,
> confirmed=true)
> Nov 08 09:35:05 [3567] h4 crmd: notice: process_lrm_event: h4-lxc_unifi_monitor_0:499
> [ unifi doesn't exist\n ]
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.3 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.4 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=4
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources:
> <lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="8:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;8:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h2" call-id="451" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h2/crmd/358,
> version=0.1449.4)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_modify operation for section status to master (origin=local/crmd/405)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.4 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.5 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=5
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources:
> <lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="14:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;14:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h4" call-id="499" rc-code="7" op-status="0" interval="0"
> last-run="1478
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h4/crmd/405,
> version=0.1449.5)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.5 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.6 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=6
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources:
> <lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
> provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="7:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;7:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h2" call-id="447" rc-code="7" op-status="0" interval="0"
> last-run="14786262
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h2/crmd/359,
> version=0.1449.6)
> Nov 08 09:35:05 [3567] h4 crmd: notice: process_lrm_event: Operation
> fs_unifi_monitor_0: not running (node=h4, call=495, rc=7, cib-update=406,
> confirmed=true)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.6 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.7 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=7
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources:
> <lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="5:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;5:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h1" call-id="449" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h1/crmd/358,
> version=0.1449.7)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_modify operation for section status to master (origin=local/crmd/406)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.7 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.8 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=8
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources:
> <lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
> provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="13:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;13:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h4" call-id="495" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h4/crmd/406,
> version=0.1449.8)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.8 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.9 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=9
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources:
> <lrm_resource id="lxc_unifi" type="lxc" class="ocf" provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="lxc_unifi_last_failure_0"
> operation_key="lxc_unifi_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
> transition-key="11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="465" rc-code="0" op-status="0" interval="0" last-r
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="lxc_unifi_last_0" operation_key="lxc_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;11:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="465" rc-code="0" op-status="0" interval="0"
> last-run="1478
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h3/crmd/802,
> version=0.1449.9)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.9 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.10 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=10
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838209']/lrm[@id='167838209']/lrm_resources:
> <lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
> provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="4:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;4:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h1" call-id="445" rc-code="7" op-status="0" interval="0"
> last-run="14786262
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h1/crmd/359,
> version=0.1449.10)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.10 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.11 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=11
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources:
> <lrm_resource id="fs_unifi" type="Filesystem" class="ocf"
> provider="heartbeat"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="fs_unifi_last_failure_0"
> operation_key="fs_unifi_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
> transition-key="10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="461" rc-code="0" op-status="0" interval="0" last-run
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="fs_unifi_last_0" operation_key="fs_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;10:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="461" rc-code="0" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h3/crmd/803,
> version=0.1449.11)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.11 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.12 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=12
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838210']/lrm[@id='167838210']/lrm_resources:
> <lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="6:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;6:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h2" call-id="443" rc-code="7" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h2/crmd/360,
> version=0.1449.12)
> Nov 08 09:35:05 [3567] h4 crmd: notice: process_lrm_event: Operation
> rbd_unifi_monitor_0: not running (node=h4, call=491, rc=7, cib-update=407,
> confirmed=true)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.12 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.13 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=13
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838211']/lrm[@id='167838211']/lrm_resources:
> <lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="rbd_unifi_last_failure_0"
> operation_key="rbd_unifi_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
> transition-key="9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="457" rc-code="0" op-status="0" interval="0" last-run
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:0;9:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h3" call-id="457" rc-code="0" op-status="0" interval="0"
> last-run="147862
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h3/crmd/804,
> version=0.1449.13)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Forwarding
> cib_modify operation for section status to master (origin=local/crmd/407)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: ---
> 0.1449.13 2
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: Diff: +++
> 0.1449.14 (null)
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: + /cib:
> @num_updates=14
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
> /cib/status/node_state[@id='167838212']/lrm[@id='167838212']/lrm_resources:
> <lrm_resource id="rbd_unifi" type="rbd" class="ocf" provider="ceph"/>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> <lrm_rsc_op id="rbd_unifi_last_0" operation_key="rbd_unifi_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.10"
> transition-key="12:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> transition-magic="0:7;12:121:7:2bc5b523-b947-4035-b296-4c099646298b"
> on_node="h4" call-id="491" rc-code="7" op-status="0" interval="0"
> last-run="1478
> Nov 08 09:35:05 [3562] h4 cib: info: cib_perform_op: ++
>
> </lrm_resource>
> Nov 08 09:35:05 [3562] h4 cib: info: cib_process_request: Completed
> cib_modify operation for section status: OK (rc=0, origin=h4/crmd/407,
> version=0.1449.14)
> Nov 08 09:35:10 [3562] h4 cib: info: cib_process_ping: Reporting
> our current digest to h3: b5a63d2ac8b4835f8c8a2f310eb6648c for 0.1449.14
> (0x12ddc90 0)
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20161108/46e48eee/attachment-0003.html>
More information about the Users
mailing list