[Pacemaker] "crm_node -f -R NodeID" remove the wrong node?
Andrew Beekhof
andrew at beekhof.net
Mon Apr 14 03:08:44 EDT 2014
On 13 Apr 2014, at 11:36 am, Caizhifeng <bluewindow at h3c.com> wrote:
> Hi all,
> I'm kinda new to Pacemaker and have some questions I hope some of you could help me with, any idea(s) would be highly appreciated, thank you.
> I'am building the HA cluster with corosync2.3.3 + cluster-glue + pacemaker-1.1.11-rc5. The problem is, when I try to remove a node from HA cluster with "crm_node -f -R nodeID", it seems to remove the wrong node.
>
> My setup is as follows(and it can be reproduced):
> 1. The HA is ok with 2 resource and 2 nodes, and with stonith-enabled="false":
> root at h1:/opt/bin# crm status
> Last updated: Sat Apr 12 16:00:51 2014
> Last change: Sat Apr 12 16:00:43 2014 via cibadmin on h1
> Stack: corosync
> Current DC: h1 (1084752017) - partition with quorum
> Version: 1.1.10-33f9d09
> 2 Nodes configured
> 2 Resources configured
>
>
> Online: [ h0 h1 ]
>
> VMdos-1 (ocf::heartbeat:VirtualDomain): Started h1
> root at h1:/opt/bin#
>
> root at h1:/opt/bin# crm_node --list
> 1084752017 h1
> 1084752016 h0
> root at h1:/opt/bin#
>
> 2. remove node h0 from HA
> (1). Stop pacemakerd and corosync service on node h0 (2). Run the follow cmd on node h1 to Remove node info from cib:
> cibadmin --delete --obj_type status --crm_xml "<node_state id=\"1084752016\"/>"
> cibadmin --delete --obj_type nodes --crm_xml "<node id=\"1084752016\"/>"
>
> at this time; the result of "crm status" shows that node " h0 " is offline and there's no information of node h0 in CIB. But the "crm_node --list " still include node" h0"
>
> 3. crm_node -f -R 1084752016
Could you re-run this command but add -VVVVVVVV please?
I suspect the fault lies in this binary.
> After this cmd, node " h1" is removed, and result of "crm status " is as follow(with 0 node):
>
> root at h1:/opt/bin# crm status
> Last updated: Sat Apr 12 15:59:42 2014
> Last change: Sat Apr 12 15:59:37 2014 via crm_node on h1
> Stack: corosync
> Current DC: NONE
> 0 Nodes configured
> 2 Resources configured
> root at h1:/opt/bin
>
>
> The corosync.conf is as follow:
> .............
> quorum {
> provider: corosync_votequorum
> expected_votes: 2
> allow_downscale: 1
> two_node: 1
> }
> ...........
>
>
> The corosync.log is as follow, it seems node " h1" is removed due to fencing!
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] ringbuffer.c:299 Free'ing ringbuffer: /dev/shm/qb-cmap-request-25004-14152-27-header
> Apr 12 16:22:35 [25026] h1 cib: debug: activateCibXml: Triggering CIB write for cib_delete op
> Apr 12 16:22:35 [25031] h1 crmd: notice: crm_reap_dead_member: Removing h0/1084752016 from the membership list
> Apr 12 16:22:35 [25031] h1 crmd: notice: reap_crm_member: Purged 1 peers with id=1084752016 and/or uname=(null) from the membership cache
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: log_cib_diff: Config update: Local-only Change: 0.12.1
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - <cib admin_epoch="0" epoch="11" num_updates="1">
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - <configuration>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - <nodes>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <node id="1084752017" uname="h1"/>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - </nodes>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - </configuration>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - </cib>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: ++ <cib epoch="12" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.8" cib-last-written="Sat Apr 12 16:22:35 2014" update-origin="h1" update-client="crm_node" have-quorum="1" dc-uuid="1084752017"/>
> Apr 12 16:22:35 [25026] h1 cib: notice: log_cib_diff: cib:diff: Local-only Change: 0.12.1
> Apr 12 16:22:35 [25026] h1 cib: notice: cib:diff: -- <node id="1084752017" uname="h1"/>
> Apr 12 16:22:35 [25026] h1 cib: notice: cib:diff: ++ <cib epoch="12" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.8" cib-last-written="Sat Apr 12 16:22:35 2014" update-origin="h1" update-client="crm_node" have-quorum="1" dc-uuid="1084752017"/>
> Apr 12 16:22:35 [25026] h1 cib: info: cib_process_request: Completed cib_delete operation for section nodes: OK (rc=0, origin=local/crm_node/2, version=0.12.1)
> Apr 12 16:22:35 [25031] h1 crmd: debug: te_update_diff: Processing diff (cib_delete): 0.11.1 -> 0.12.1 (S_IDLE)
> Apr 12 16:22:35 [25031] h1 crmd: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, node=, tag=diff, id=(null), magic=NA, cib=0.12.1) : Non-status change
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <diff crm_feature_set="3.0.8" digest="3cccef06483ac4dfeadfb562f6f8293a">
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <diff-removed admin_epoch="0" epoch="11" num_updates="1">
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <cib admin_epoch="0" epoch="11" num_updates="1">
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <configuration>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <nodes>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <node id="1084752017" uname="h1" __crm_diff_marker__="removed:top"/>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause </nodes>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause </configuration>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause </cib>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause </diff-removed>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <diff-added>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <cib epoch="12" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.8" cib-last-written="Sat Apr 12 16:22:35 2014" update-origin="h1" update-client="crm_node" have-quorum="1" dc-uuid="1084752017"/>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause </diff-added>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause </diff>
> Apr 12 16:22:35 [25031] h1 crmd: debug: s_crmd_fsa: Processing I_PE_CALC: [ state=S_IDLE cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Apr 12 16:22:35 [25031] h1 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Apr 12 16:22:35 [25031] h1 crmd: debug: do_state_transition: All 1 cluster nodes are eligible to run resources.
> Apr 12 16:22:35 [25031] h1 crmd: debug: do_pe_invoke: Query 38: Requesting the current CIB: S_POLICY_ENGINE
> Apr 12 16:22:35 [25026] h1 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/38, version=0.12.1)
> Apr 12 16:22:35 [25031] h1 crmd: debug: do_pe_invoke_callback: Invoking the PE: query=38, ref=pe_calc-dc-1397290955-21, seq=1024, quorate=1
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: Diff: --- 0.12.1
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: Diff: +++ 0.12.2 3d673c27c3c92939b41c7207edee9f46
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - <cib num_updates="1">
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - <status>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <node_state id="1084752017" uname="h1" in_ccm="true" crmd="online" crm-debug-origin="post_cache_update" join="member" expected="member">
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <transient_attributes id="1084752017">
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <instance_attributes id="status-1084752017">
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <nvpair id="status-1084752017-shutdown" name="shutdown" value="0"/>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <nvpair id="status-1084752017-probe_complete" name="probe_complete" value="true"/>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- </instance_attributes>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- </transient_attributes>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <lrm id="1084752017">
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <lrm_resources>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <lrm_resource id="VMdos-1" type="VirtualDomain" class="ocf" provider="heartbeat">
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <lrm_rsc_op id="VMdos-1_last_failure_0" operation_key="VMdos-1_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.8" transition-key="6:4:7:affc1dba-30eb-458d-8f86-c37d0268e52c" transition-magic="0:0;6:4:7:affc1dba-30eb-458d-8f86-c37d0268e52c" call-id="5" rc-code="0" op-status="0" interval="0" last-run="1397290607" last-rc-change="1397290607" exec-time="92" queue-time="0" op-digest="
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <lrm_rsc_op id="VMdos-1_monitor_30000" operation_key="VMdos-1_monitor_30000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.8" transition-key="7:5:0:affc1dba-30eb-458d-8f86-c37d0268e52c" transition-magic="0:0;7:5:0:affc1dba-30eb-458d-8f86-c37d0268e52c" call-id="6" rc-code="0" op-status="0" interval="30000" last-rc-change="1397290607" exec-time="67" queue-time="0" op-digest="0874c7ce5f61a12
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- </lrm_resource>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <lrm_resource id="VMdos-2" type="VirtualDomain" class="ocf" provider="heartbeat">
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- <lrm_rsc_op id="VMdos-2_last_0" operation_key="VMdos-2_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.8" transition-key="7:6:7:affc1dba-30eb-458d-8f86-c37d0268e52c" transition-magic="0:7;7:6:7:affc1dba-30eb-458d-8f86-c37d0268e52c" call-id="10" rc-code="7" op-status="0" interval="0" last-run="1397290608" last-rc-change="1397290608" exec-time="61" queue-time="0" op-digest="c7d22be
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- </lrm_resource>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- </lrm_resources>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- </lrm>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: -- </node_state>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - </status>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: - </cib>
> Apr 12 16:22:35 [25027] h1 stonith-ng: debug: Config update: ++ <cib epoch="12" num_updates="2" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.8" cib-last-written="Sat Apr 12 16:22:35 2014" update-origin="h1" update-client="crm_node" have-quorum="1" dc-uuid="1084752017"/>
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: STONITH timeout: 60000
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: STONITH of failed nodes is disabled
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: Stop all active resources: false
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: Default stickiness: 0
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: On loss of CCM Quorum: Stop ALL resources
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_domains: Unpacking domains
> Apr 12 16:22:35 [25030] h1 pengine: warning: unpack_status: Node h1 in status section no longer exists
> Apr 12 16:22:35 [25030] h1 pengine: info: unpack_status: Node 1084752017 is unknown
> Apr 12 16:22:35 [25030] h1 pengine: info: native_print: VMdos-1 (ocf::heartbeat:VirtualDomain): Stopped
> Apr 12 16:22:35 [25030] h1 pengine: info: native_print: VMdos-2 (ocf::heartbeat:VirtualDomain): Stopped
> Apr 12 16:22:35 [25030] h1 pengine: debug: native_assign_node: Could not allocate a node for VMdos-1
> Apr 12 16:22:35 [25030] h1 pengine: info: native_color: Resource VMdos-1 cannot run anywhere
> Apr 12 16:22:35 [25030] h1 pengine: debug: native_assign_node: Could not allocate a node for VMdos-2
> Apr 12 16:22:35 [25030] h1 pengine: info: native_color: Resource VMdos-2 cannot run anywhere
> Apr 12 16:22:35 [25030] h1 pengine: info: LogActions: Leave VMdos-1 (Stopped)
> Apr 12 16:22:35 [25030] h1 pengine: info: LogActions: Leave VMdos-2 (Stopped)
> Apr 12 16:22:35 [25030] h1 pengine: notice: process_pe_message: Calculated Transition 4: /var/lib/pacemaker/pengine/pe-input-4.bz2
> Apr 12 16:22:35 [25031] h1 crmd: debug: s_crmd_fsa: Processing I_PE_SUCCESS: [ state=S_POLICY_ENGINE cause=C_IPC_MESSAGE origin=handle_response ]
> Apr 12 16:22:35 [25031] h1 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
> Apr 12 16:22:35 [25031] h1 crmd: debug: unpack_graph: Unpacked transition 4: 0 actions in 0 synapses
> Apr 12 16:22:35 [25031] h1 crmd: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1397290955-21) derived from /var/lib/pacemaker/pengine/pe-input-4.bz2
> Apr 12 16:22:35 [25031] h1 crmd: debug: print_graph: Empty transition graph
> Apr 12 16:22:35 [25026] h1 cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=local/crm_node/3, version=0.12.2)
> Apr 12 16:22:35 [25031] h1 crmd: debug: te_update_diff: Processing diff (cib_delete): 0.12.1 -> 0.12.2 (S_TRANSITION_ENGINE)
> Apr 12 16:22:35 [25031] h1 crmd: info: abort_transition_graph: te_update_diff:188 - Triggered transition abort (complete=0, node=h1, tag=transient_attributes, id=1084752017, magic=NA, cib=0.12.2) : Transient attribute: removal
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <transient_attributes id="1084752017">
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <instance_attributes id="status-1084752017">
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <nvpair id="status-1084752017-shutdown" name="shutdown" value="0"/>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause <nvpair id="status-1084752017-probe_complete" name="probe_complete" value="true"/>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause </instance_attributes>
> Apr 12 16:22:35 [25031] h1 crmd: debug: abort_transition_graph: Cause </transient_attributes>
> Apr 12 16:22:35 [25031] h1 crmd: debug: update_abort_priority: Abort priority upgraded from 0 to 1000000
> Apr 12 16:22:35 [25031] h1 crmd: debug: update_abort_priority: Abort action done superceeded by restart
> Apr 12 16:22:35 [25031] h1 crmd: notice: run_graph: Transition 4 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-4.bz2): Complete
> Apr 12 16:22:35 [25031] h1 crmd: debug: print_graph: Empty transition graph
> Apr 12 16:22:35 [25031] h1 crmd: debug: te_graph_trigger: Transition 4 is now complete
> Apr 12 16:22:35 [25031] h1 crmd: debug: notify_crmd: Processing transition completion in state S_TRANSITION_ENGINE
> Apr 12 16:22:35 [25031] h1 crmd: debug: notify_crmd: Transition 4 status: restart - Transient attribute: removal
> Apr 12 16:22:35 [25031] h1 crmd: debug: s_crmd_fsa: Processing I_PE_CALC: [ state=S_TRANSITION_ENGINE cause=C_FSA_INTERNAL origin=notify_crmd ]
> Apr 12 16:22:35 [25031] h1 crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
> Apr 12 16:22:35 [25031] h1 crmd: debug: do_state_transition: All 1 cluster nodes are eligible to run resources.
> Apr 12 16:22:35 [25031] h1 crmd: debug: do_pe_invoke: Query 39: Requesting the current CIB: S_POLICY_ENGINE
> Apr 12 16:22:35 [25024] h1 pacemakerd: info: crm_client_new: Connecting 0x25b4ea0 for uid=0 gid=0 pid=14152 id=f3612e17-0806-4355-a3fc-2cf1feda1e6d
> Apr 12 16:22:35 [25024] h1 pacemakerd: debug: handle_new_connection: IPC credentials authenticated (25024-14152-10)
> Apr 12 16:22:35 [25024] h1 pacemakerd: debug: qb_ipcs_shm_connect: connecting to client [14152]
> Apr 12 16:22:35 [25026] h1 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/39, version=0.12.2)
> Apr 12 16:22:35 [25024] h1 pacemakerd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792
> Apr 12 16:22:35 [25026] h1 cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (25026-14152-13)
> Apr 12 16:22:35 [25026] h1 cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(25026-14152-13) state:2
> Apr 12 16:22:35 [25026] h1 cib: info: crm_client_destroy: Destroying 0 events
> Apr 12 16:22:35 [25026] h1 cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-25026-14152-13-header
> Apr 12 16:22:35 [25026] h1 cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-25026-14152-13-header
> Apr 12 16:22:35 [25024] h1 pacemakerd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792
> Apr 12 16:22:35 [25024] h1 pacemakerd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] ipc_setup.c:478 IPC credentials authenticated (25004-14152-27)
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] ipc_shm.c:294 connecting to client [14152]
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] ringbuffer.c:236 shm size:1048589; real_size:1052672; rb->word_size:263168
> Apr 12 16:22:35 [25031] h1 crmd: debug: do_pe_invoke_callback: Invoking the PE: query=39, ref=pe_calc-dc-1397290955-22, seq=1024, quorate=1
> Apr 12 16:22:35 [25031] h1 crmd: debug: qb_ipcs_dispatch_connection_request: HUP conn (25031-14152-14)
> Apr 12 16:22:35 [25031] h1 crmd: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(25031-14152-14) state:2
> Apr 12 16:22:35 [25031] h1 crmd: info: crm_client_destroy: Destroying 0 events
> Apr 12 16:22:35 [25031] h1 crmd: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-crmd-response-25031-14152-14-header
> ...
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: STONITH timeout: 60000
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: STONITH of failed nodes is disabled
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: Stop all active resources: false
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: Default stickiness: 0
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: On loss of CCM Quorum: Stop ALL resources
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
> Apr 12 16:22:35 [25030] h1 pengine: debug: unpack_domains: Unpacking domains
> Apr 12 16:22:35 [25030] h1 pengine: info: native_print: VMdos-1 (ocf::heartbeat:VirtualDomain): Stopped
> Apr 12 16:22:35 [25030] h1 pengine: info: native_print: VMdos-2 (ocf::heartbeat:VirtualDomain): Stopped
> Apr 12 16:22:35 [25030] h1 pengine: debug: native_assign_node: Could not allocate a node for VMdos-1
> Apr 12 16:22:35 [25030] h1 pengine: info: native_color: Resource VMdos-1 cannot run anywhere
> Apr 12 16:22:35 [25030] h1 pengine: debug: native_assign_node: Could not allocate a node for VMdos-2
> Apr 12 16:22:35 [25030] h1 pengine: info: native_color: Resource VMdos-2 cannot run anywhere
> Apr 12 16:22:35 [25030] h1 pengine: info: LogActions: Leave VMdos-1 (Stopped)
> Apr 12 16:22:35 [25030] h1 pengine: info: LogActions: Leave VMdos-2 (Stopped)
> Apr 12 16:22:35 [25030] h1 pengine: notice: process_pe_message: Calculated Transition 5: /var/lib/pacemaker/pengine/pe-input-5.bz2
> Apr 12 16:22:35 [25031] h1 crmd: debug: s_crmd_fsa: Processing I_PE_SUCCESS: [ state=S_POLICY_ENGINE cause=C_IPC_MESSAGE origin=handle_response ]
> Apr 12 16:22:35 [25031] h1 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
> Apr 12 16:22:35 [25031] h1 crmd: debug: unpack_graph: Unpacked transition 5: 0 actions in 0 synapses
> Apr 12 16:22:35 [25031] h1 crmd: info: do_te_invoke: Processing graph 5 (ref=pe_calc-dc-1397290955-22) derived from /var/lib/pacemaker/pengine/pe-input-5.bz2
> Apr 12 16:22:35 [25031] h1 crmd: debug: print_graph: Empty transition graph
> Apr 12 16:22:35 [25031] h1 crmd: notice: run_graph: Transition 5 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-5.bz2): Complete
> Apr 12 16:22:35 [25031] h1 crmd: debug: print_graph: Empty transition graph
> Apr 12 16:22:35 [25031] h1 crmd: debug: te_graph_trigger: Transition 5 is now complete
> Apr 12 16:22:35 [25031] h1 crmd: debug: notify_crmd: Processing transition completion in state S_TRANSITION_ENGINE
> Apr 12 16:22:35 [25031] h1 crmd: debug: notify_crmd: Transition 5 status: done - <null>
> Apr 12 16:22:35 [25031] h1 crmd: debug: s_crmd_fsa: Processing I_TE_SUCCESS: [ state=S_TRANSITION_ENGINE cause=C_FSA_INTERNAL origin=notify_crmd ]
> Apr 12 16:22:35 [25031] h1 crmd: info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
> Apr 12 16:22:35 [25031] h1 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
> Apr 12 16:22:35 [25031] h1 crmd: debug: do_state_transition: Starting PEngine Recheck Timer
> Apr 12 16:22:35 [25031] h1 crmd: debug: crm_timer_start: Started PEngine Recheck Timer (I_PE_CALC:900000ms), src=62
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] ringbuffer.c:236 shm size:1048589; real_size:1052672; rb->word_size:263168
> Apr 12 16:22:35 [25026] h1 cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-9.raw
> Apr 12 16:22:35 [25026] h1 cib: debug: write_cib_contents: Writing CIB to disk
> Apr 12 16:22:35 [25026] h1 cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-25026-14152-13-header
> Apr 12 16:22:35 [25026] h1 cib: info: write_cib_contents: Wrote version 0.12.0 of the CIB to disk (digest: bd7d26226d6aa75f28b9eb670a67e944)
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] ringbuffer.c:236 shm size:1048589; real_size:1052672; rb->word_size:263168
> Apr 12 16:22:35 [25003] h1 corosync debug [MAIN ] ipc_glue.c:272 connection created
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] cmap.c:306 lib_init_fn: conn=0x7f32d0f4f4b0
> Apr 12 16:22:35 [25024] h1 pacemakerd: notice: pcmk_ipc_dispatch: Instructing peers to remove references to node (null)/1084752016
> Apr 12 16:22:35 [25026] h1 cib: info: crm_client_new: Connecting 0xa23770 for uid=0 gid=0 pid=14152 id=701dfa3c-d585-49d2-bd1e-f44636823e1b
> Apr 12 16:22:35 [25026] h1 cib: debug: handle_new_connection: IPC credentials authenticated (25026-14152-13)
> Apr 12 16:22:35 [25026] h1 cib: debug: qb_ipcs_shm_connect: connecting to client [14152]
> Apr 12 16:22:35 [25026] h1 cib: debug: write_cib_contents: Wrote digest bd7d26226d6aa75f28b9eb670a67e944 to disk
> Apr 12 16:22:35 [25026] h1 cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.y6wnAg (digest: /var/lib/pacemaker/cib/cib.ncXFvs)
> ...
> Apr 12 16:22:35 [25026] h1 cib: debug: write_cib_contents: Activating /var/lib/pacemaker/cib/cib.y6wnAg
> Apr 12 16:22:35 [25026] h1 cib: info: cib_process_request: Completed cib_delete operation for section nodes: OK (rc=0, origin=local/crm_node/2, version=0.12.2)
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] ipcs.c:757 HUP conn (25004-14152-27)
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] ipcs.c:605 qb_ipcs_disconnect(25004-14152-27) state:2
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] loop_poll_epoll.c:117 epoll_ctl(del): Bad file descriptor (9)
> Apr 12 16:22:35 [25003] h1 corosync debug [MAIN ] ipc_glue.c:417 cs_ipcs_connection_closed()
> Apr 12 16:22:35 [25003] h1 corosync debug [QB ] cmap.c:325 exit_fn for conn=0x7f32d0f4f4b0
> Apr 12 16:22:35 [25003] h1 corosync debug [MAIN ] ipc_glue.c:390 cs_ipcs_connection_destroyed()
> Apr 12 16:22:35 [25026] h1 cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=local/crm_node/3, version=0.12.2)
> ...
> Apr 12 16:22:35 [25024] h1 pacemakerd: debug: qb_ipcs_dispatch_connection_request: HUP conn (25024-14152-10)
> Apr 12 16:22:35 [25024] h1 pacemakerd: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(25024-14152-10) state:2
> Apr 12 16:22:35 [25024] h1 pacemakerd: info: crm_client_destroy: Destroying 0 events
> ...
> Apr 12 16:22:35 [25026] h1 cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (25026-14152-13)
> Apr 12 16:22:35 [25026] h1 cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(25026-14152-13) state:2
> Apr 12 16:22:35 [25026] h1 cib: info: crm_client_destroy: Destroying 0 events
> ...
> Apr 12 16:22:35 [25024] h1 pacemakerd: notice: crm_reap_dead_member: Removing h0/1084752016 from the membership list
> Apr 12 16:22:35 [25024] h1 pacemakerd: notice: reap_crm_member: Purged 1 peers with id=1084752016 and/or uname=(null) from the membership cache
>
>
> -------------------------------------------------------------------------------------------------------------------------------------
> 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
> 邮件！
> This e-mail and its attachments contain confidential information from H3C, which is
> intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140414/08232fbb/attachment-0003.sig>
More information about the Pacemaker
mailing list