[ClusterLabs] How to avoid stopping ordered resources on cleanup?
Ken Gaillot
kgaillot at redhat.com
Fri Sep 15 11:58:11 EDT 2017
On Fri, 2017-09-15 at 15:23 +0000, CART Andreas wrote:
> Hello
>
> I think there is a more general misunderstanding on my side.
> I reproduced the “problem” with a very simple test-cluster containing
> just 2 dummy resources:
>
> Pacemaker Nodes:
> deneb682 deneb683
>
> Resources:
> Resource: Res1 (class=ocf provider=pacemaker type=Dummy)
> Operations: start interval=0s timeout=20 (Res1-start-interval-0s)
> stop interval=0s timeout=20 (Res1-stop-interval-0s)
> monitor interval=10 timeout=20 (Res1-monitor-interval-
> 10)
> Resource: Res2 (class=ocf provider=pacemaker type=Dummy)
> Operations: start interval=0s timeout=20 (Res2-start-interval-0s)
> stop interval=0s timeout=20 (Res2-stop-interval-0s)
> monitor interval=10 timeout=20 (Res2-monitor-interval-
> 10)
>
> Ordering Constraints:
> start Res1 then start Res2 (kind:Mandatory) (id:order-Res1-Res2-
> mandatory)
>
> Cluster Properties:
> cluster-infrastructure: cman
> default-resource-stickiness: 100
> no-quorum-policy: ignore
> symmetric-cluster: true
>
> When I call “pcs resource cleanup Res1” this will result in an
> interruption of service at the side of Res2 (i.e. stop Res2 …)
> My – unconfirmed – assumption was, that pacemaker would first detect
> the current state of the resource(s) by calling monitor and then
> decide if there are any actions to be performed.
> But from reading the logfiles I would interpret that Res1 is
> temporarily removed from the cib and re-inserted again. And this
> results in stopping Res2 until Res1 has confirmed state “started”.
Correct, removing the resource's operation history is how pacemaker
triggers a re-probe of the current status.
> As I interpret the documentation it would be possible to avoid this
> behaviour by configuring the order constraint with kind=Optional.
> But I am not sure if this would result in any other undeserved side
> effects. (e.g on reverse order when stopping)
kind=Optional constraints only apply when both actions need to be done
in the same transition. I.e. if a single cluster check finds that both
Res1 and Res2 need to be started, Res1 will be started before Res2. But
it is entirely possible that Res2 can be started in an earlier
transition, with Res1 still stopped, and a later transition starts
Res1. Similarly when stopping, Res2 will be stopped first, if both need
to be stopped.
In your original scenario, if your master/slave resource will only bind
to the IP after it is up, kind=Optional won't be reliable. But if the
master/slave resource binds to the wildcard IP, then the order really
doesn't matter -- you could keep the colocation constraint and drop the
ordering.
> Another work-around seems to be setting the dependent resource to
> unmanaged, perform the cleanup and then set it back to managed.
This is what i would recommend, if you have to keep the mandatory
ordering.
> And I wonder if “pcs resource failcount reset” would do the trick
> WITHOUT any actions being performed if no change in state is
> necessary.
> But I think to remember that we already tried this now and then and
> sometimes such a failed resource was not started after the failcount
> reset. (But I am not sure and had not yet time to try to reproduce.)
No, in newer pacemaker versions, crm_failcount --delete is equivalent
to a crm_resource --cleanup. (pcs calls these to actually perform the
work)
> Is there any deeper insight which might help with a sound
> understanding of this issue?
It's a side effect of the current CIB implementation. Pacemaker's
policy engine determines the current state of a resource by checking
its operation history in the CIB. Cleanups remove the operation
history, thus making the current state unknown, forcing a re-probe. As
a side effect, any dependencies no longer have their constraints
satisfied until the re-probe completes.
It would be theoretically possible to implement a "cleanup old
failures" option that would clear a resource's fail count and remove
only its operation history entries for failed operations, as long as
doing so does not change the current state determination. But that
would be quite complicated, and setting the resource unmanaged is an
easy workaround.
> Kind regards
> Andreas Cart
> From: Klaus Wenninger [mailto:kwenning at redhat.com]
> Sent: Mittwoch, 13. September 2017 13:33
> To: Cluster Labs - All topics related to open-source clustering
> welcomed <users at clusterlabs.org>; CART Andreas <andreas.cart at sonorys.
> at>
> Subject: Re: [ClusterLabs] How to avoid stopping ordered resources on
> cleanup?
>
> On 09/13/2017 10:26 AM, CART Andreas wrote:
> Hello
>
> We have a basic 2 node active/passive cluster with Pacemaker (1.1.16
> , pcs: 0.9.148) / CMAN (3.0.12.1) / Corosync (1.4.7) on RHEL 6.8.
>
> On the occasion of testing the cluster we noticed that dependent
> resources are stopped when calling cleanup for a resource lower down
> the order chain.
> But for the production system we have to avoid this by all means.
>
> Actually we have a master/slave resource and dependent on that and
> colocated with the master a clustered IP address.
> Here is the relevant part of the cluster config:
> Cluster Name: bam-cluster
> Corosync Nodes:
> bam1-backend bam2-backend
> Pacemaker Nodes:
> bam1-backend bam2-backend
>
> Resources:
> Master: mvno-100-master
> Meta Attrs: master-node-max=1 clone-max=2 notify=true master-max=1
> clone-node-max=1
> Resource: mvno-100 (class=ocf provider=sonorys type=sba)
> Operations: start interval=0s timeout=120s (mvno-100-start-
> interval-0s)
> stop interval=0s timeout=120s (mvno-100-stop-interval-
> 0s)
> promote interval=0s timeout=5s (mvno-100-promote-
> interval-0s)
> demote interval=0s timeout=5s (mvno-100-demote-
> interval-0s)
> monitor interval=9 role=Master (mvno-100-monitor-
> interval-9)
> monitor interval=10 role=Slave (mvno-100-monitor-
> interval-10)
> Resource: IPaddrAdminToolMvno100 (class=ocf provider=heartbeat
> type=IPaddr2)
> Attributes: ip=xxx.xxx.xxx.xxx cidr_netmask=yy
> Operations: start interval=0s timeout=20s (IPaddrAdminToolMvno100-
> start-interval-0s)
> stop interval=0s timeout=20s (IPaddrAdminToolMvno100-
> stop-interval-0s)
> monitor interval=10s timeout=20s
> (IPaddrAdminToolMvno100-monitor-interval-10s)
> Ordering Constraints:
> start IPaddrAdminToolMvno100 then promote mvno-100-master
> (kind:Mandatory) (id:order-IPaddrAdminToolMvno100-mvno-100-master-
> mandatory)
> Colocation Constraints:
> IPaddrAdminToolMvno100 with mvno-100-master (score:INFINITY) (rsc-
> role:Started) (with-rsc-role:Master) (id:colocation-
> IPaddrAdminToolMvno100-mvno-100-master-INFINITY)
> Cluster Properties:
> cluster-infrastructure: cman
> default-resource-stickiness: 100
> no-quorum-policy: ignore
> symmetric-cluster: true
>
> Have you tried the other way round?
> like: promote mvno-100 then start IPaddrAdminToolMvno100
>
> Regards,
> Klaus
>
>
>
> For example we had the following failure notice:
> Failed Actions:
> * mvno-100_monitor_10000 on bam1-backend 'not running' (7): call=40,
> status=complete, exitreason='none',
> last-rc-change='Tue Sep 12 09:55:49 2017', queued=0ms, exec=0ms
>
> And cleared it by calling:
> # pcs resource cleanup mvno-100
> Wed Sep 13 06:43:53 UTC 2017
> Cleaning up mvno-100:0 on bam1-backend, removing fail-count-mvno-100
> Cleaning up mvno-100:0 on bam2-backend, removing fail-count-mvno-100
> Waiting for 2 replies from the CRMd.. OK
>
> This resulted in stopping IPaddrAdminToolMvno100 and starting it
> again, which actually means a (short) interruption of service … which
> we would like to avoid.
>
> Here is the corresponding logfile for the cluster:
> Sep 13 06:43:53 [3829] bam1-omc attrd: notice:
> attrd_cs_dispatch: Update relayed from bam2-backend
> Sep 13 06:43:53 [3829] bam1-omc attrd: notice:
> attrd_trigger_update: Sending flush op to all hosts for: fail-
> count-mvno-100 (<null>)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.167.0 2
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.167.1 590cae41f6fad2b00a8cf067302756e4
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: -- /cib/status/node_state[@id='bam1-
> backend']/lrm[@id='bam1-
> backend']/lrm_resources/lrm_resource[@id='mvno-100']
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=1
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='bam1-backend']//lrm_resource[@id='mvno-10
> 0']: OK (rc=0, origin=local/crmd/1485, version=0.167.0)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.167.0 2
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.167.1 (null)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: -- /cib/status/node_state[@id='bam2-
> backend']/lrm[@id='bam2-
> backend']/lrm_resources/lrm_resource[@id='mvno-100']
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=1
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='bam2-backend']//lrm_resource[@id='mvno-10
> 0']: OK (rc=0, origin=bam2-backend/crmd/39, version=0.167.0)
> Sep 13 06:43:53 [3831] bam1-omc crmd: info:
> delete_resource: Removing resource mvno-100 for 1c0903b8-1dcc-432a-
> b6a8-e409ce917dda (root) on bam2-backend
> Sep 13 06:43:53 [3828] bam1-omc lrmd: info:
> cancel_recurring_action: Cancelling ocf operation mvno-
> 100_monitor_9000
> Sep 13 06:43:53 [3829] bam1-omc attrd: notice:
> attrd_perform_update: Sent delete 57: node=bam1-backend,
> attr=fail-count-mvno-100, id=<n/a>, set=(null), section=status
> Sep 13 06:43:53 [3831] bam1-omc crmd: info:
> lrm_remove_deleted_op: Removing op mvno-100_monitor_9000:95 for
> deleted resource mvno-100
> Sep 13 06:43:53 [3831] bam1-omc crmd: info:
> notify_deleted: Notifying 1c0903b8-1dcc-432a-b6a8-e409ce917dda on
> bam2-backend that mvno-100 was deleted
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.167.0 2
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.167.1 (null)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: -- /cib/status/node_state[@id='bam1-
> backend']/transient_attributes[@id='bam1-
> backend']/instance_attributes[@id='s
> tatus-bam1-backend']/nvpair[@id='status-bam1-backend-fail-count-mvno-
> 100']
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=1
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_delete operation for section
> status: OK (rc=0, origin=local/attrd/57, version=0.167.1)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.167.1 2
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.167.2 (null)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: -- /cib/status/node_state[@id='bam1-
> backend']/lrm[@id='bam1-
> backend']/lrm_resources/lrm_resource[@id='mvno-100']
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=2
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='bam1-backend']//lrm_resource[@id='mvno-10
> 0']: OK (rc=0, origin=local/crmd/1486, version=0.167.2)
> Sep 13 06:43:53 [3831] bam1-omc crmd: info:
> abort_transition_graph: Transition aborted by deletion of
> nvpair[@id='status-bam1-backend-fail-count-mvno-100']: Transient
> attrib
> ute change | cib=0.167.1 source=abort_unless_down:345
> path=/cib/status/node_state[@id='bam1-
> backend']/transient_attributes[@id='bam1-
> backend']/instance_attributes[@id='status-bam1-backe
> nd']/nvpair[@id='status-bam1-backend-fail-count-mvno-100']
> complete=true
> Sep 13 06:43:53 [3831] bam1-omc crmd: info:
> abort_transition_graph: Transition aborted by deletion of
> lrm_resource[@id='mvno-100']: Resource state removal | cib=0.167.2
> sour
> ce=abort_unless_down:345 path=/cib/status/node_state[@id='bam1-
> backend']/lrm[@id='bam1-
> backend']/lrm_resources/lrm_resource[@id='mvno-100'] complete=true
> Sep 13 06:43:53 [3828] bam1-omc lrmd: info:
> process_lrmd_get_rsc_info: Resource 'mvno-100' not found (3
> active resources)
> Sep 13 06:43:53 [3831] bam1-omc crmd: info:
> process_lrm_event: Result of monitor operation for mvno-100 on
> bam1-backend: Cancelled | call=95 key=mvno-100_monitor_9000 c
> onfirmed=true
> Sep 13 06:43:53 [3831] bam1-omc crmd: info:
> update_history_cache: Resource mvno-100 no longer exists, not
> updating cache
> Sep 13 06:43:53 [3831] bam1-omc crmd: notice:
> do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE
> | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition
> _graph
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.167.2 2
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.0 (null)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @epoch=168, @num_updates=0
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: +
> /cib/configuration/crm_config/cluster_property_set[@id='cib-
> bootstrap-options']/nvpair[@id='cib-bootstrap-opti
> ons-last-lrm-refresh']: @value=1505285033
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=local/crmd/1488, version=0.168.0)
> Sep 13 06:43:53 [3831] bam1-omc crmd: info:
> abort_transition_graph: Transition aborted by cib-bootstrap-
> options-last-lrm-refresh doing modify last-lrm-refresh=1505285033:
> Configuration change | cib=0.168.0 source=te_update_diff:444
> path=/cib/configuration/crm_config/cluster_property_set[@id='cib-
> bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-
> refresh'] complete=true
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_file_backup: Archived previous version as
> /var/lib/pacemaker/cib/cib-65.raw
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.168.0 2
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.1 (null)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: -- /cib/status/node_state[@id='bam2-
> backend']/lrm[@id='bam2-
> backend']/lrm_resources/lrm_resource[@id='mvno-100']
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=1
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='bam2-backend']//lrm_resource[@id='mvno-100']: OK
> (rc=0, origin=bam2-backend/crmd/40, version=0.168.1)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> crm_config: OK (rc=0, origin=bam2-backend/crmd/42, version=0.168.1)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_file_write_with_digest: Wrote version 0.168.0 of the CIB to
> disk (digest: 7d2e14eee5a440883e9659536e70260f)
> Sep 13 06:43:53 [3826] bam1-omc cib: info:
> cib_file_write_with_digest: Reading cluster configuration file
> /var/lib/pacemaker/cib/cib.BuFPbI (digest:
> /var/lib/pacemaker/cib/cib.jHzYAc)
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> parse_notifications: No optional alerts section in cib
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> abort_transition_graph: Transition aborted by deletion of
> lrm_resource[@id='mvno-100']: Resource state removal | cib=0.168.1
> source=abort_unless_down:345 path=/cib/status/node_state[@id='bam2-
> backend']/lrm[@id='bam2-
> backend']/lrm_resources/lrm_resource[@id='mvno-100'] complete=true
> Sep 13 06:43:54 [3830] bam1-omc pengine: notice:
> unpack_config: On loss of CCM Quorum: Ignore
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> determine_online_status_fencing: Node bam2-backend is active
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> determine_online_status: Node bam2-backend is online
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> determine_online_status_fencing: Node bam1-backend is active
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> determine_online_status: Node bam1-backend is online
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> determine_op_status: Operation monitor found resource
> IPaddrAdminToolMvno100 active on bam1-backend
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> clone_print: Master/Slave Set: mvno-100-master [mvno-100]
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> short_print: Stopped: [ bam1-backend bam2-backend ]
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> native_print: IPaddrAdminToolMvno100
> (ocf::heartbeat:IPaddr2): Started bam1-backend
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> native_print: IPMI_fence_bam1 (stonith:fence_ipmilan):
> Started bam2-backend
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> native_print: IPMI_fence_bam2 (stonith:fence_ipmilan):
> Started bam1-backend
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> master_color: mvno-100-master: Promoted 0 instances of a possible
> 1 to master
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> native_color: Resource IPaddrAdminToolMvno100 cannot run anywhere
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> RecurringOp: Start recurring monitor (10s) for mvno-100:0 on
> bam1-backend
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> RecurringOp: Start recurring monitor (10s) for mvno-100:1 on
> bam2-backend
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> RecurringOp: Start recurring monitor (10s) for mvno-100:0 on
> bam1-backend
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> RecurringOp: Start recurring monitor (10s) for mvno-100:1 on
> bam2-backend
> Sep 13 06:43:54 [3830] bam1-omc pengine: notice:
> LogActions: Start mvno-100:0 (bam1-backend)
> Sep 13 06:43:54 [3830] bam1-omc pengine: notice:
> LogActions: Start mvno-100:1 (bam2-backend)
> Sep 13 06:43:54 [3830] bam1-omc pengine: notice:
> LogActions: Stop IPaddrAdminToolMvno100 (bam1-backend)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> I suspect the cluster “forgets” all status information about the
> cleaned up resource and therefore supposes it is in stopped state and
> has to be started.
> And due to the order constraint the dependent IP address has to be
> stopped as long as the cluster does not see the first resource in
> state started.
> Shouldn’t there be a monitor operation to determine the actual state
> before performing any actions based on unconfirmed assumptions of
> state?
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> LogActions: Leave IPMI_fence_bam1 (Started bam2-backend)
> Sep 13 06:43:54 [3830] bam1-omc pengine: info:
> LogActions: Leave IPMI_fence_bam2 (Started bam1-backend)
> Sep 13 06:43:54 [3830] bam1-omc pengine: notice:
> process_pe_message: Calculated transition 1379, saving inputs
> in /var/lib/pacemaker/pengine/pe-input-31.bz2
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> do_state_transition: State transition S_POLICY_ENGINE ->
> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE
> origin=handle_response
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> do_te_invoke: Processing graph 1379 (ref=pe_calc-dc-1505285034-
> 1562) derived from /var/lib/pacemaker/pengine/pe-input-31.bz2
> Sep 13 06:43:54 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating monitor operation mvno-100:0_monitor_0
> locally on bam1-backend | action 5
> Sep 13 06:43:54 [3828] bam1-omc lrmd: info:
> process_lrmd_get_rsc_info: Resource 'mvno-100' not found (3
> active resources)
> Sep 13 06:43:54 [3828] bam1-omc lrmd: info:
> process_lrmd_get_rsc_info: Resource 'mvno-100:0' not found (3
> active resources)
> Sep 13 06:43:54 [3828] bam1-omc lrmd: info:
> process_lrmd_rsc_register: Added 'mvno-100' to the rsc list (4
> active resources)
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> do_lrm_rsc_op: Performing key=5:1379:7:e2c19428-0707-4677-a89a-
> ff1c19ebe57c op=mvno-100_monitor_0
> Sep 13 06:43:54 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating monitor operation mvno-100:1_monitor_0
> on bam2-backend | action 6
> Sep 13 06:43:54 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating stop operation
> IPaddrAdminToolMvno100_stop_0 locally on bam1-backend | action 35
> Sep 13 06:43:54 [3828] bam1-omc lrmd: info:
> cancel_recurring_action: Cancelling ocf operation
> IPaddrAdminToolMvno100_monitor_10000
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> do_lrm_rsc_op: Performing key=35:1379:0:e2c19428-0707-4677-a89a-
> ff1c19ebe57c op=IPaddrAdminToolMvno100_stop_0
> Sep 13 06:43:54 [3828] bam1-omc lrmd: info:
> log_execute: executing - rsc:IPaddrAdminToolMvno100 action:stop
> call_id:104
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> process_lrm_event: Result of monitor operation for
> IPaddrAdminToolMvno100 on bam1-backend: Cancelled | call=90
> key=IPaddrAdminToolMvno100_monitor_10000 confirmed=true
> sba(mvno-100)[56850]: 2017/09/13_06:43:54 INFO: mvno-100 monitor
> started
> sba(mvno-100)[56850]: 2017/09/13_06:43:54 INFO: Check status
> IPaddr2(IPaddrAdminToolMvno100)[56851]: 2017/09/13_06:43:54 INFO: IP
> status = ok, IP_CIP=
> Sep 13 06:43:54 [3828] bam1-omc lrmd: info:
> log_finished: finished - rsc:IPaddrAdminToolMvno100 action:stop
> call_id:104 pid:56851 exit-code:0 exec-time:72ms queue-time:0ms
> Sep 13 06:43:54 [3831] bam1-omc crmd: notice:
> process_lrm_event: Result of stop operation for
> IPaddrAdminToolMvno100 on bam1-backend: 0 (ok) | call=104
> key=IPaddrAdminToolMvno100_stop_0 confirmed=true cib-update=1494
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.168.1 2
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.2 (null)
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=2
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib/status/node_state[@id='bam1-
> backend']/lrm[@id='bam1-
> backend']/lrm_resources/lrm_resource[@id='IPaddrAdminToolMvno100']/lr
> m_rsc_op[@id='IPaddrAdminToolMvno100_last_0']: @operation_key=IPaddrA
> dminToolMvno100_stop_0, @operation=stop, @transition-
> key=35:1379:0:e2c19428-0707-4677-a89a-ff1c19ebe57c, @transition-
> magic=0:0;35:1379:0:e2c19428-0707-4677-a89a-ff1c19ebe57c, @call-
> id=104, @last-run=1505285034, @last-rc-change=1505285034, @ex
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=local/crmd/1494, version=0.168.2)
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> match_graph_event: Action IPaddrAdminToolMvno100_stop_0 (35)
> confirmed on bam1-backend (rc=0)
> sba(mvno-100)[56850]: 2017/09/13_06:43:54 INFO: mvno-100 monitor
> returned 8
> Sep 13 06:43:54 [3831] bam1-omc crmd: notice:
> process_lrm_event: Result of probe operation for mvno-100 on
> bam1-backend: 8 (master) | call=102 key=mvno-100_monitor_0
> confirmed=true cib-update=1495
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.168.2 2
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.3 (null)
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=3
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op: ++ /cib/status/node_state[@id='bam1-
> backend']/lrm[@id='bam1-backend']/lrm_resources: <lrm_resource
> id="mvno-100" type="sba" class="ocf" provider="sonorys"/>
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op:
> ++
> <lrm_rsc_op id="mvno-100_last_failure_0"
> operation_key="mvno-100_monitor_0" operation="monitor" crm-debug-
> origin="do_update_resource" crm_feature_set="3.0.11" transition-
> key="5:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c" transition-
> magic="0:8;5:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c"
> on_node="bam1-backend" call-id="102" rc-code="8" op-status="0"
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op:
> ++
> <lrm_rsc_op id="mvno-100_last_0"
> operation_key="mvno-100_monitor_0" operation="monitor" crm-debug-
> origin="do_update_resource" crm_feature_set="3.0.11" transition-
> key="5:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c" transition-
> magic="0:8;5:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c"
> on_node="bam1-backend" call-id="102" rc-code="8" op-status="0"
> interva
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_perform_op:
> ++
> </lrm_resource>
> Sep 13 06:43:54 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=local/crmd/1495, version=0.168.3)
> Sep 13 06:43:54 [3831] bam1-omc crmd: warning:
> status_from_rc: Action 5 (mvno-100:0_monitor_0) on bam1-backend
> failed (target: 7 vs. rc: 8): Error
> Sep 13 06:43:54 [3831] bam1-omc crmd: notice:
> abort_transition_graph: Transition aborted by operation mvno-
> 100_monitor_0 'create' on bam1-backend: Event failed |
> magic=0:8;5:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c cib=0.168.3
> source=match_graph_event:310 complete=false
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_monitor_0 (5) confirmed on
> bam1-backend (rc=8)
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> process_graph_event: Detected action (1379.5) mvno-
> 100_monitor_0.102=master: failed
> Sep 13 06:43:54 [3831] bam1-omc crmd: warning:
> status_from_rc: Action 5 (mvno-100:0_monitor_0) on bam1-backend
> failed (target: 7 vs. rc: 8): Error
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> abort_transition_graph: Transition aborted by operation mvno-
> 100_monitor_0 'create' on bam1-backend: Event failed |
> magic=0:8;5:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c cib=0.168.3
> source=match_graph_event:310 complete=false
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_monitor_0 (5) confirmed on
> bam1-backend (rc=8)
> Sep 13 06:43:54 [3831] bam1-omc crmd: info:
> process_graph_event: Detected action (1379.5) mvno-
> 100_monitor_0.102=master: failed
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.168.3 2
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.4 (null)
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=4
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: ++ /cib/status/node_state[@id='bam2-
> backend']/lrm[@id='bam2-backend']/lrm_resources: <lrm_resource
> id="mvno-100" type="sba" class="ocf" provider="sonorys"/>
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op:
> ++
> <lrm_rsc_op id="mvno-100_last_failure_0"
> operation_key="mvno-100_monitor_0" operation="monitor" crm-debug-
> origin="do_update_resource" crm_feature_set="3.0.11" transition-
> key="6:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c" transition-
> magic="0:0;6:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c"
> on_node="bam2-backend" call-id="82" rc-code="0" op-status="0"
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op:
> ++
> <lrm_rsc_op id="mvno-100_last_0"
> operation_key="mvno-100_monitor_0" operation="monitor" crm-debug-
> origin="do_update_resource" crm_feature_set="3.0.11" transition-
> key="6:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c" transition-
> magic="0:0;6:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c"
> on_node="bam2-backend" call-id="82" rc-code="0" op-status="0"
> interval
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op:
> ++
> </lrm_resource>
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=bam2-backend/crmd/44, version=0.168.4)
> Sep 13 06:43:55 [3831] bam1-omc crmd: warning:
> status_from_rc: Action 6 (mvno-100:1_monitor_0) on bam2-backend
> failed (target: 7 vs. rc: 0): Error
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> abort_transition_graph: Transition aborted by operation mvno-
> 100_monitor_0 'create' on bam2-backend: Event failed |
> magic=0:0;6:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c cib=0.168.4
> source=match_graph_event:310 complete=false
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_monitor_0 (6) confirmed on
> bam2-backend (rc=0)
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> process_graph_event: Detected action (1379.6) mvno-
> 100_monitor_0.82=ok: failed
> Sep 13 06:43:55 [3831] bam1-omc crmd: warning:
> status_from_rc: Action 6 (mvno-100:1_monitor_0) on bam2-backend
> failed (target: 7 vs. rc: 0): Error
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> abort_transition_graph: Transition aborted by operation mvno-
> 100_monitor_0 'create' on bam2-backend: Event failed |
> magic=0:0;6:1379:7:e2c19428-0707-4677-a89a-ff1c19ebe57c cib=0.168.4
> source=match_graph_event:310 complete=false
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_monitor_0 (6) confirmed on
> bam2-backend (rc=0)
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> process_graph_event: Detected action (1379.6) mvno-
> 100_monitor_0.82=ok: failed
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> run_graph: Transition 1379 (Complete=6, Pending=0, Fired=0,
> Skipped=0, Incomplete=10, Source=/var/lib/pacemaker/pengine/pe-input-
> 31.bz2): Complete
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> do_state_transition: State transition S_TRANSITION_ENGINE ->
> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=notify_crmd
> Sep 13 06:43:55 [3830] bam1-omc pengine: notice:
> unpack_config: On loss of CCM Quorum: Ignore
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> determine_online_status_fencing: Node bam2-backend is active
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> determine_online_status: Node bam2-backend is online
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> determine_online_status_fencing: Node bam1-backend is active
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> determine_online_status: Node bam1-backend is online
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> determine_op_status: Operation monitor found resource mvno-100:0
> active on bam2-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> determine_op_status: Operation monitor found resource mvno-100:0
> active on bam2-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> determine_op_status: Operation monitor found resource
> IPaddrAdminToolMvno100 active on bam1-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> determine_op_status: Operation monitor found resource mvno-100:1
> active in master mode on bam1-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> determine_op_status: Operation monitor found resource mvno-100:1
> active in master mode on bam1-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> clone_print: Master/Slave Set: mvno-100-master [mvno-100]
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> short_print: Masters: [ bam1-backend ]
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> short_print: Slaves: [ bam2-backend ]
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> native_print: IPaddrAdminToolMvno100
> (ocf::heartbeat:IPaddr2): Stopped
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> native_print: IPMI_fence_bam1 (stonith:fence_ipmilan):
> Started bam2-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> native_print: IPMI_fence_bam2 (stonith:fence_ipmilan):
> Started bam1-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> master_color: Promoting mvno-100:1 (Master bam1-backend)
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> master_color: mvno-100-master: Promoted 1 instances of a possible
> 1 to master
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> RecurringOp: Start recurring monitor (10s) for mvno-100:0 on
> bam2-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> RecurringOp: Start recurring monitor (9s) for mvno-100:1 on
> bam1-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> RecurringOp: Start recurring monitor (10s) for mvno-100:0 on
> bam2-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> RecurringOp: Start recurring monitor (9s) for mvno-100:1 on
> bam1-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> RecurringOp: Start recurring monitor (10s) for
> IPaddrAdminToolMvno100 on bam1-backend
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> LogActions: Leave mvno-100:0 (Slave bam2-backend)
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> LogActions: Leave mvno-100:1 (Master bam1-backend)
> Sep 13 06:43:55 [3830] bam1-omc pengine: notice:
> LogActions: Start IPaddrAdminToolMvno100 (bam1-backend)
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> LogActions: Leave IPMI_fence_bam1 (Started bam2-backend)
> Sep 13 06:43:55 [3830] bam1-omc pengine: info:
> LogActions: Leave IPMI_fence_bam2 (Started bam1-backend)
> Sep 13 06:43:55 [3830] bam1-omc pengine: notice:
> process_pe_message: Calculated transition 1380, saving inputs
> in /var/lib/pacemaker/pengine/pe-input-32.bz2
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> do_state_transition: State transition S_POLICY_ENGINE ->
> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE
> origin=handle_response
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> do_te_invoke: Processing graph 1380 (ref=pe_calc-dc-1505285035-
> 1566) derived from /var/lib/pacemaker/pengine/pe-input-32.bz2
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating start operation
> IPaddrAdminToolMvno100_start_0 locally on bam1-backend | action 36
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> do_lrm_rsc_op: Performing key=36:1380:0:e2c19428-0707-4677-a89a-
> ff1c19ebe57c op=IPaddrAdminToolMvno100_start_0
> Sep 13 06:43:55 [3828] bam1-omc lrmd: info:
> log_execute: executing - rsc:IPaddrAdminToolMvno100 action:start
> call_id:105
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating notify operation mvno-
> 100_pre_notify_promote_0 on bam2-backend | action 52
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating notify operation mvno-
> 100_pre_notify_promote_0 locally on bam1-backend | action 54
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> do_lrm_rsc_op: Performing key=54:1380:0:e2c19428-0707-4677-a89a-
> ff1c19ebe57c op=mvno-100_notify_0
> Sep 13 06:43:55 [3828] bam1-omc lrmd: info:
> log_execute: executing - rsc:mvno-100 action:notify call_id:106
> sba(mvno-100)[56964]: 2017/09/13_06:43:55 INFO: mvno-100 notify
> started
> Sep 13 06:43:55 [3828] bam1-omc lrmd: info:
> log_finished: finished - rsc:mvno-100 action:notify call_id:106
> pid:56964 exit-code:3 exec-time:21ms queue-time:0ms
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_notify_0 (54) confirmed on
> bam1-backend (rc=0)
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> process_lrm_event: Result of notify operation for mvno-100 on
> bam1-backend: 0 (ok) | call=106 key=mvno-100_notify_0 confirmed=true
> cib-update=0
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_notify_0 (52) confirmed on
> bam2-backend (rc=0)
> IPaddr2(IPaddrAdminToolMvno100)[56963]: 2017/09/13_06:43:55 INFO:
> Adding inet address xxx.xxx.xxx.xxx/yy with broadcast address
> zzz.zzz.zzz.zzz to device bond2
> IPaddr2(IPaddrAdminToolMvno100)[56963]: 2017/09/13_06:43:55 INFO:
> Bringing device bond2 up
> IPaddr2(IPaddrAdminToolMvno100)[56963]: 2017/09/13_06:43:55 INFO:
> /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-
> agents/send_arp-xxx.xxx.xxx.xxx bond2 xxx.xxx.xxx.xxx auto not_used
> not_used
> Sep 13 06:43:55 [3828] bam1-omc lrmd: info:
> log_finished: finished - rsc:IPaddrAdminToolMvno100 action:start
> call_id:105 pid:56963 exit-code:0 exec-time:91ms queue-time:0ms
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> action_synced_wait: Managed IPaddr2_meta-data_0 process 57044
> exited with rc=0
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> process_lrm_event: Result of start operation for
> IPaddrAdminToolMvno100 on bam1-backend: 0 (ok) | call=105
> key=IPaddrAdminToolMvno100_start_0 confirmed=true cib-update=1497
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.168.4 2
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.5 (null)
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=5
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib/status/node_state[@id='bam1-
> backend']/lrm[@id='bam1-
> backend']/lrm_resources/lrm_resource[@id='IPaddrAdminToolMvno100']/lr
> m_rsc_op[@id='IPaddrAdminToolMvno100_last_0']: @operation_key=IPaddrA
> dminToolMvno100_start_0, @operation=start, @transition-
> key=36:1380:0:e2c19428-0707-4677-a89a-ff1c19ebe57c, @transition-
> magic=0:0;36:1380:0:e2c19428-0707-4677-a89a-ff1c19ebe57c, @call-
> id=105, @last-run=1505285035, @last-rc-change=1505285035, @
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=local/crmd/1497, version=0.168.5)
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action IPaddrAdminToolMvno100_start_0 (36)
> confirmed on bam1-backend (rc=0)
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating monitor operation
> IPaddrAdminToolMvno100_monitor_10000 locally on bam1-backend | action
> 37
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> do_lrm_rsc_op: Performing key=37:1380:0:e2c19428-0707-4677-a89a-
> ff1c19ebe57c op=IPaddrAdminToolMvno100_monitor_10000
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating promote operation mvno-100_promote_0
> locally on bam1-backend | action 10
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> do_lrm_rsc_op: Performing key=10:1380:0:e2c19428-0707-4677-a89a-
> ff1c19ebe57c op=mvno-100_promote_0
> Sep 13 06:43:55 [3828] bam1-omc lrmd: info:
> log_execute: executing - rsc:mvno-100 action:promote call_id:108
> sba(mvno-100)[57049]: 2017/09/13_06:43:55 INFO: mvno-100 promote
> started
> sba(mvno-100)[57049]: 2017/09/13_06:43:55 INFO: Check status
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> process_lrm_event: Result of monitor operation for
> IPaddrAdminToolMvno100 on bam1-backend: 0 (ok) | call=107
> key=IPaddrAdminToolMvno100_monitor_10000 confirmed=false cib-
> update=1498
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.168.5 2
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.6 (null)
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=6
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib/status/node_state[@id='bam1-
> backend']/lrm[@id='bam1-
> backend']/lrm_resources/lrm_resource[@id='IPaddrAdminToolMvno100']/lr
> m_rsc_op[@id='IPaddrAdminToolMvno100_monitor_10000']: @transition-
> key=37:1380:0:e2c19428-0707-4677-a89a-ff1c19ebe57c, @transition-
> magic=0:0;37:1380:0:e2c19428-0707-4677-a89a-ff1c19ebe57c, @call-
> id=107, @last-rc-change=1505285035, @exec-time=65
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=local/crmd/1498, version=0.168.6)
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action IPaddrAdminToolMvno100_monitor_10000
> (37) confirmed on bam1-backend (rc=0)
> sba(mvno-100)[57049]: 2017/09/13_06:43:55 INFO: Resource is already
> running as Master
> sba(mvno-100)[57049]: 2017/09/13_06:43:55 INFO: mvno-100 promote
> returned 0
> Sep 13 06:43:55 [3828] bam1-omc lrmd: info:
> log_finished: finished - rsc:mvno-100 action:promote call_id:108
> pid:57049 exit-code:0 exec-time:287ms queue-time:0ms
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> process_lrm_event: Result of promote operation for mvno-100 on
> bam1-backend: 0 (ok) | call=108 key=mvno-100_promote_0 confirmed=true
> cib-update=1499
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.168.6 2
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.7 (null)
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=7
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib/status/node_state[@id='bam1-
> backend']/lrm[@id='bam1-
> backend']/lrm_resources/lrm_resource[@id='mvno-
> 100']/lrm_rsc_op[@id='mvno-100_last_0']: @operation_key=mvno-
> 100_promote_0, @operation=promote, @transition-
> key=10:1380:0:e2c19428-0707-4677-a89a-ff1c19ebe57c, @transition-
> magic=0:0;10:1380:0:e2c19428-0707-4677-a89a-ff1c19ebe57c, @call-
> id=108, @rc-code=0, @last-run=1505285035, @last-rc-change=1505285035,
> @exec-time=287
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=local/crmd/1499, version=0.168.7)
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_promote_0 (10) confirmed on
> bam1-backend (rc=0)
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating notify operation mvno-
> 100_post_notify_promote_0 on bam2-backend | action 53
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating notify operation mvno-
> 100_post_notify_promote_0 locally on bam1-backend | action 55
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> do_lrm_rsc_op: Performing key=55:1380:0:e2c19428-0707-4677-a89a-
> ff1c19ebe57c op=mvno-100_notify_0
> Sep 13 06:43:55 [3828] bam1-omc lrmd: info:
> log_execute: executing - rsc:mvno-100 action:notify call_id:109
> sba(mvno-100)[57157]: 2017/09/13_06:43:55 INFO: mvno-100 notify
> started
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_notify_0 (53) confirmed on
> bam2-backend (rc=0)
> Sep 13 06:43:55 [3828] bam1-omc lrmd: info:
> log_finished: finished - rsc:mvno-100 action:notify call_id:109
> pid:57157 exit-code:3 exec-time:25ms queue-time:0ms
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_notify_0 (55) confirmed on
> bam1-backend (rc=0)
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> process_lrm_event: Result of notify operation for mvno-100 on
> bam1-backend: 0 (ok) | call=109 key=mvno-100_notify_0 confirmed=true
> cib-update=0
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating monitor operation mvno-100_monitor_10000
> on bam2-backend | action 6
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> te_rsc_command: Initiating monitor operation mvno-100_monitor_9000
> locally on bam1-backend | action 11
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> do_lrm_rsc_op: Performing key=11:1380:8:e2c19428-0707-4677-a89a-
> ff1c19ebe57c op=mvno-100_monitor_9000
> sba(mvno-100)[57166]: 2017/09/13_06:43:55 INFO: mvno-100 monitor
> started
> sba(mvno-100)[57166]: 2017/09/13_06:43:55 INFO: Check status
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.168.7 2
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.8 (null)
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=8
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: ++ /cib/status/node_state[@id='bam2-
> backend']/lrm[@id='bam2-
> backend']/lrm_resources/lrm_resource[@id='mvno-100']: <lrm_rsc_op
> id="mvno-100_monitor_10000" operation_key="mvno-100_monitor_10000"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.11" transition-key="6:1380:0:e2c19428-0707-4677-
> a89a-ff1c19ebe57c" transition-magic="0:0;6:1380:0:e2c19428-0707-4677-
> a89a-ff1c19ebe57c" on_node="bam2-backend" call-id="
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=bam2-backend/crmd/45, version=0.168.8)
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_monitor_10000 (6) confirmed
> on bam2-backend (rc=0)
> sba(mvno-100)[57166]: 2017/09/13_06:43:55 INFO: mvno-100 monitor
> returned 8
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> process_lrm_event: Result of monitor operation for mvno-100 on
> bam1-backend: 8 (master) | call=110 key=mvno-100_monitor_9000
> confirmed=false cib-update=1500
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: --- 0.168.8 2
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: Diff: +++ 0.168.9 (null)
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: + /cib: @num_updates=9
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_perform_op: ++ /cib/status/node_state[@id='bam1-
> backend']/lrm[@id='bam1-
> backend']/lrm_resources/lrm_resource[@id='mvno-100']: <lrm_rsc_op
> id="mvno-100_monitor_9000" operation_key="mvno-100_monitor_9000"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.11" transition-key="11:1380:8:e2c19428-0707-
> 4677-a89a-ff1c19ebe57c" transition-magic="0:8;11:1380:8:e2c19428-
> 0707-4677-a89a-ff1c19ebe57c" on_node="bam1-backend" call-id="
> Sep 13 06:43:55 [3826] bam1-omc cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=local/crmd/1500, version=0.168.9)
> Sep 13 06:43:55 [3831] bam1-omc crmd: info:
> match_graph_event: Action mvno-100_monitor_9000 (11) confirmed
> on bam1-backend (rc=8)
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> run_graph: Transition 1380 (Complete=15, Pending=0, Fired=0,
> Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-
> 32.bz2): Complete
> Sep 13 06:43:55 [3831] bam1-omc crmd: info: do_log: Input
> I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
> Sep 13 06:43:55 [3831] bam1-omc crmd: notice:
> do_state_transition: State transition S_TRANSITION_ENGINE ->
> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
>
> How can we avoid this interruption of service on cleanup?
>
> Kind regards
> Andreas Cart
>
>
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
--
Ken Gaillot <kgaillot at redhat.com>
More information about the Users
mailing list