[Pacemaker] Stonith issue with fence_virsh
Andrew Beekhof
andrew at beekhof.net
Thu Oct 31 00:09:03 UTC 2013
Personally I use fence_xvm. IIRC, it's the supported equivalent of fence_virsh.
On 24 Oct 2013, at 6:38 pm, Beo Banks <beo.banks at googlemail.com> wrote:
> hi,
>
> i have enable the debug option and i use the ip instead of hostname
>
> primitive stonith-zarafa02 stonith:fence_virsh \
> params pcmk_host_list="zarafa02" pcmk_host_check="static-list" action="reboot" ipaddr="*.*.*.*" secure="true" login="root" identity_file="/root/.ssh/id_rsa" debug="/var/log/stonith.log" verbose="true" \
>
>
> it seems that the fence_virsh can establish the connection but he don´t start the reboot command via pacemaker
> is there any option to find out how pacemaker runs the command?
>
> /var/log/stonith.log
> [EXPECT]# virsh list --all
> Id Name State
> ----------------------------------------------------
> 16 zarafa02 running
>
> [EXPECT]# ^C
>
>
>
> it should be (i saw it via comandline)
>
> EXPECT]# virsh list --all
> Id Name State
> ----------------------------------------------------
> 16 zarafa02 running
>
> [EXPECT]# virsh destroy zarafa02
> Domain zarafa02 destroyed
>
> [EXPECT]# virsh list --all
> Id Name State
> ----------------------------------------------------
> - zarafa02 shut off
>
> [EXPECT]# virsh start zarafa02
> Domain zarafa02 started
>
> [EXPECT]# virsh list --all
> Id Name State
> ----------------------------------------------------
> 17 zarafa02 running
>
> [EXPECT]# Success: Rebooted
>
>
>
>
>
> 2013/10/23 Beo Banks <beo.banks at googlemail.com>
> i use /etc/hosts
>
> ping host01 is correct.
>
>
> maybe that´s the reason i test the fence_virsh as root.
>
> primitive stonith-zarafa02 stonith:fence_virsh \
> params pcmk_host_list="zarafa02" pcmk_host_check="static-list" action="reboot" ipaddr="host02" secure="true" login="root" identity_file="/root/.ssh/id_rsa" \
>
> op monitor interval="300s" \
> op start interval="0" timeout="60s" \
> meta failure-timeout="180s"
>
>
> as which user fence pacemaker the device? as root?
>
> via commanline (works)
> node1# fence_virsh -a host02 -l root -x -k /root/.ssh/id_rsa -o reboot -v -n zarafa02
>
> is there a mistake in my config?
>
>
>
>
>
> Oct 23 13:23:54 zarafa02 corosync[1700]: [MAIN ] Corosync Cluster Engine ('1.4.1'): started and ready to provide service.
> Oct 23 13:23:54 zarafa02 corosync[1700]: [MAIN ] Corosync built-in features: nss dbus rdma snmp
> Oct 23 13:23:54 zarafa02 corosync[1700]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
> Oct 23 13:23:54 zarafa02 corosync[1700]: [TOTEM ] Initializing transport (UDP/IP Multicast).
> Oct 23 13:23:54 zarafa02 corosync[1700]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Oct 23 13:23:54 zarafa02 corosync[1700]: [TOTEM ] Initializing transport (UDP/IP Multicast).
> Oct 23 13:23:54 zarafa02 corosync[1700]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Oct 23 13:23:55 zarafa02 corosync[1700]: [TOTEM ] The network interface [10.0.0.22] is now up.
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: process_ais_conf: Reading configure
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] ERROR: process_ais_conf: You have configured a cluster using the Pacemaker plugin for Corosync. The plugin is not supported in this environment and will be removed very soon.
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] ERROR: process_ais_conf: Please see Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) for details on using Pacemaker with CMAN
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: config_find_init: Local handle: 5880381755227111425 for logging
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: config_find_next: Processing additional logging options...
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: Found 'off' for option: debug
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: Found 'yes' for option: to_logfile
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: Found '/var/log/cluster/corosync.log' for option: logfile
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: Found 'yes' for option: to_syslog
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: config_find_init: Local handle: 4835695805891346434 for quorum
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: config_find_next: No additional configuration supplied for: quorum
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: No default for option: provider
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: config_find_init: Local handle: 4552499517957603331 for service
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: config_find_next: Processing additional service options...
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: Found '1' for option: ver
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: process_ais_conf: Enabling MCP mode: Use the Pacemaker init script to complete Pacemaker startup
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: pcmk_startup: CRM: Initialized
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] Logging: Initialized pcmk_startup
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: pcmk_startup: Service: 9
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: pcmk_startup: Local hostname: zarafa02
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: pcmk_update_nodeid: Local node id: 369098762
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: update_member: Creating entry for node 369098762 born on 0
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: update_member: 0x141afc0 Node 369098762 now known as zarafa02 (was: (null))
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: update_member: Node zarafa02 now has 1 quorum votes (was 0)
> Oct 23 13:23:55 zarafa02 corosync[1700]: [pcmk ] info: update_member: Node 369098762/zarafa02 is now: member
> Oct 23 13:23:55 zarafa02 corosync[1700]: [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.1.8
> Oct 23 13:23:55 zarafa02 corosync[1700]: [SERV ] Service engine loaded: corosync extended virtual synchrony service
> Oct 23 13:23:55 zarafa02 corosync[1700]: [SERV ] Service engine loaded: corosync configuration service
> Oct 23 13:23:55 zarafa02 corosync[1700]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
> Oct 23 13:23:55 zarafa02 corosync[1700]: [SERV ] Service engine loaded: corosync cluster config database access v1.01
> Oct 23 13:23:55 zarafa02 corosync[1700]: [SERV ] Service engine loaded: corosync profile loading service
> Oct 23 13:23:55 zarafa02 corosync[1700]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1
> Oct 23 13:23:55 zarafa02 corosync[1700]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
> Oct 23 13:23:55 zarafa02 corosync[1700]: [TOTEM ] The network interface [0.0.0.0] is now up.
> Oct 23 13:23:56 zarafa02 corosync[1700]: [TOTEM ] Incrementing problem counter for seqid 1 iface 0.0.0.0to [1 of 10]
> Oct 23 13:23:56 zarafa02 corosync[1700]: [TOTEM ] Incrementing problem counter for seqid 1 iface 0.0.0.0to [1 of 10]
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 268: memb=0, new=0, lost=0
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 268: memb=1, new=1, lost=0
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: pcmk_peer_update: NEW: zarafa02 369098762
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: pcmk_peer_update: MEMB: zarafa02 369098762
> Oct 23 13:23:56 zarafa02 corosync[1700]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
> Oct 23 13:23:56 zarafa02 corosync[1700]: [CPG ] chosen downlist: sender r(0) ip(10.0.0.22) r(1) ip(0.0.0.0) ; members(old:0 left:0)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [MAIN ] Completed service synchronization, ready to provide service.
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 272: memb=1, new=0, lost=0
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: pcmk_peer_update: memb: zarafa02 369098762
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 272: memb=2, new=1, lost=0
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: update_member: Creating entry for node 352321546 born on 272
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: update_member: Node 352321546/unknown is now: member
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 352321546
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 352321546
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: pcmk_peer_update: MEMB: zarafa02 369098762
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: send_member_notification: Sending membership update 272 to 0 children
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: update_member: 0x141afc0 Node 369098762 ((null)) born on: 272
> Oct 23 13:23:56 zarafa02 corosync[1700]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: update_member: 0x14247d0 Node 352321546 (zarafa01) born on: 248
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: update_member: 0x14247d0 Node 352321546 now known as zarafa01 (was: (null))
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: update_member: Node zarafa01 now has 1 quorum votes (was 0)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] info: send_member_notification: Sending membership update 272 to 0 children
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [CPG ] chosen downlist: sender r(0) ip(10.0.0.21) r(1) ip(0.0.0.0) ; members(old:1 left:0)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [MAIN ] Completed service synchronization, ready to provide service.
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:23:56 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:23:58 zarafa02 corosync[1700]: [TOTEM ] ring 1 active with no faults(1)
> Oct 23 13:23:59 zarafa02 kernel: drbd: initialized. Version: 8.3.15 (api:88/proto:86-97)
> Oct 23 13:23:59 zarafa02 kernel: drbd: GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil at Build64R6, 2012-12-20 20:09:51
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: Starting worker thread (from cqueue [1987])
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: disk( Diskless -> Attaching )
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: Found 4 transactions (11 active extents) in activity log.
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: Method to ensure write ordering: flush
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: max BIO size = 131072
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: drbd_bm_resize called with capacity == 524270872
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: resync bitmap: bits=65533859 words=1023967 pages=2000
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: size = 250 GB (262135436 KB)
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: bitmap READ of 2000 pages took 11 jiffies
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: recounting of set bits took additional 8 jiffies
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: disk( Attaching -> Consistent )
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: attached to UUIDs 455F3C39E924BE3A:0000000000000000:6F4C52EE2A21D036:6F4B52EE2A21D037
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: Starting worker thread (from cqueue [1987])
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: disk( Diskless -> Attaching )
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: Found 4 transactions (6 active extents) in activity log.
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: Method to ensure write ordering: flush
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: max BIO size = 131072
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: drbd_bm_resize called with capacity == 1258251896
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: resync bitmap: bits=157281487 words=2457524 pages=4800
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: size = 600 GB (629125948 KB)
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: bitmap READ of 4800 pages took 26 jiffies
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: recounting of set bits took additional 21 jiffies
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: disk( Attaching -> Consistent )
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: attached to UUIDs C3F30C77423EA3D4:0000000000000000:6120BBFBBB30210C:611FBBFBBB30210D
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: conn( StandAlone -> Unconnected )
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: Starting receiver thread (from drbd0_worker [1998])
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: receiver (re)started
> Oct 23 13:23:59 zarafa02 kernel: block drbd0: conn( Unconnected -> WFConnection )
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: conn( StandAlone -> Unconnected )
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: Starting receiver thread (from drbd1_worker [2017])
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: receiver (re)started
> Oct 23 13:23:59 zarafa02 kernel: block drbd1: conn( Unconnected -> WFConnection )
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: Handshake successful: Agreed network protocol version 97
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: conn( WFConnection -> WFReportParams )
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: Starting asender thread (from drbd0_receiver [2033])
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: data-integrity-alg: <not-used>
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: drbd_sync_handshake:
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: self 455F3C39E924BE3A:0000000000000000:6F4C52EE2A21D036:6F4B52EE2A21D037 bits:0 flags:0
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: peer 857412ABB0CE0739:455F3C39E924BE3B:6F4C52EE2A21D037:6F4B52EE2A21D037 bits:0 flags:0
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: uuid_compare()=-1 by rule 50
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent -> Outdated ) pdsk( DUnknown -> UpToDate )
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: Handshake successful: Agreed network protocol version 97
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: conn( WFConnection -> WFReportParams )
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: Starting asender thread (from drbd1_receiver [2037])
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: data-integrity-alg: <not-used>
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: drbd_sync_handshake:
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: self C3F30C77423EA3D4:0000000000000000:6120BBFBBB30210C:611FBBFBBB30210D bits:0 flags:0
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: peer 227D17F2074428C9:C3F30C77423EA3D5:6120BBFBBB30210D:611FBBFBBB30210D bits:0 flags:0
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: uuid_compare()=-1 by rule 50
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent -> Outdated ) pdsk( DUnknown -> UpToDate )
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID )
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: updated sync uuid 45603C39E924BE3A:0000000000000000:6F4C52EE2A21D036:6F4B52EE2A21D037
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
> ct 23 13:24:00 zarafa02 kernel: block drbd0: Began resync as SyncTarget (will sync 0 KB [0 bits set]).
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: Resync done (total 1 sec; paused 0 sec; 0 K/sec)
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: updated UUIDs 857412ABB0CE0738:0000000000000000:45603C39E924BE3A:455F3C39E924BE3B
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0
> Oct 23 13:24:00 zarafa02 crm-unfence-peer.sh[2108]: invoked for mysql
> Oct 23 13:24:00 zarafa02 cibadmin[2112]: notice: crm_log_args: Invoked: cibadmin -Ql
> Oct 23 13:24:00 zarafa02 crm-unfence-peer.sh[2108]: Could not establish cib_rw connection: Connection refused (111)
> Oct 23 13:24:00 zarafa02 crm-unfence-peer.sh[2108]: Signon to CIB failed: Transport endpoint is not connected
> Oct 23 13:24:00 zarafa02 crm-unfence-peer.sh[2108]: Init failed, could not perform requested operations
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 1 (0x100)
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: bitmap WRITE of 2000 pages took 20 jiffies
> Oct 23 13:24:00 zarafa02 kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: conn( WFBitMapT -> WFSyncUUID )
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: updated sync uuid C3F40C77423EA3D4:0000000000000000:6120BBFBBB30210C:611FBBFBBB30210D
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: helper command: /sbin/drbdadm before-resync-target minor-1
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: helper command: /sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: Began resync as SyncTarget (will sync 0 KB [0 bits set]).
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: Resync done (total 1 sec; paused 0 sec; 0 K/sec)
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: updated UUIDs 227D17F2074428C8:0000000000000000:C3F40C77423EA3D4:C3F30C77423EA3D5
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: helper command: /sbin/drbdadm after-resync-target minor-1
> Oct 23 13:24:00 zarafa02 crm-unfence-peer.sh[2121]: invoked for zarafa
> Oct 23 13:24:00 zarafa02 cibadmin[2129]: notice: crm_log_args: Invoked: cibadmin -Ql
> Oct 23 13:24:00 zarafa02 crm-unfence-peer.sh[2121]: Could not establish cib_rw connection: Connection refused (111)
> Oct 23 13:24:00 zarafa02 crm-unfence-peer.sh[2121]: Signon to CIB failed: Transport endpoint is not connected
> Oct 23 13:24:00 zarafa02 crm-unfence-peer.sh[2121]: Init failed, could not perform requested operations
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: helper command: /sbin/drbdadm after-resync-target minor-1 exit code 1 (0x100)
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: bitmap WRITE of 4800 pages took 183 jiffies
> Oct 23 13:24:00 zarafa02 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> Oct 23 13:24:00 zarafa02 abrtd: Init complete, entering main loop
> Oct 23 13:24:01 zarafa02 pacemakerd[2252]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
> Oct 23 13:24:01 zarafa02 pacemakerd[2252]: notice: main: Starting Pacemaker 1.1.8-7.el6 (Build: 394e906): generated-manpages agent-manpages ascii-docs publican-docs ncurses libqb-logging libqb-ipc corosync-plugin cman
> Oct 23 13:24:01 zarafa02 pacemakerd[2252]: notice: get_local_node_name: Defaulting to uname(2).nodename for the local classic openais (with plugin) node name
> Oct 23 13:24:01 zarafa02 pacemakerd[2252]: notice: update_node_processes: 0x8e76f0 Node 369098762 now known as zarafa02, was:
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 pacemakerd[2252]: notice: update_node_processes: 0x8e82a0 Node 352321546 now known as zarafa01, was:
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 cib[2258]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
> Oct 23 13:24:01 zarafa02 stonith-ng[2259]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
> Oct 23 13:24:01 zarafa02 stonith-ng[2259]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
> Oct 23 13:24:01 zarafa02 pengine[2262]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
> Oct 23 13:24:01 zarafa02 attrd[2261]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] info: pcmk_ipc: Recorded connection 0x1430ee0 for stonith-ng/0
> Oct 23 13:24:01 zarafa02 attrd[2261]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
> Oct 23 13:24:01 zarafa02 lrmd[2260]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] info: pcmk_ipc: Recorded connection 0x1430ee0 for stonith-ng/0
> Oct 23 13:24:01 zarafa02 attrd[2261]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
> Oct 23 13:24:01 zarafa02 lrmd[2260]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] info: pcmk_ipc: Recorded connection 0x1435260 for attrd/0
> Oct 23 13:24:01 zarafa02 crmd[2263]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
> Oct 23 13:24:01 zarafa02 crmd[2263]: notice: main: CRM Git Version: 394e906
> Oct 23 13:24:01 zarafa02 attrd[2261]: notice: main: Starting mainloop...
> Oct 23 13:24:01 zarafa02 cib[2258]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] info: pcmk_ipc: Recorded connection 0x14395e0 for cib/0
> Oct 23 13:24:01 zarafa02 corosync[1700]: [pcmk ] info: pcmk_ipc: Sending membership update 272 to cib
> Oct 23 13:24:01 zarafa02 cib[2258]: notice: ais_dispatch_message: Membership 272: quorum acquired
> Oct 23 13:24:01 zarafa02 cib[2258]: notice: crm_update_peer_state: crm_update_ais_node: Node zarafa01[352321546] - state is now member
> Oct 23 13:24:01 zarafa02 cib[2258]: notice: crm_update_peer_state: crm_update_ais_node: Node zarafa02[369098762] - state is now member
> Oct 23 13:24:02 zarafa02 stonith-ng[2259]: notice: setup_cib: Watching for stonith topology changes
> Oct 23 13:24:02 zarafa02 crmd[2263]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
> Oct 23 13:24:02 zarafa02 corosync[1700]: [pcmk ] info: pcmk_ipc: Recorded connection 0x1440080 for crmd/0
> Oct 23 13:24:02 zarafa02 corosync[1700]: [pcmk ] info: pcmk_ipc: Sending membership update 272 to crmd
> Oct 23 13:24:02 zarafa02 crmd[2263]: notice: ais_dispatch_message: Membership 272: quorum acquired
> Oct 23 13:24:02 zarafa02 crmd[2263]: notice: crm_update_peer_state: crm_update_ais_node: Node zarafa01[352321546] - state is now member
> Oct 23 13:24:02 zarafa02 crmd[2263]: notice: crm_update_peer_state: crm_update_ais_node: Node zarafa02[369098762] - state is now member
> Oct 23 13:24:02 zarafa02 crmd[2263]: notice: do_started: The local CRM is operational
> Oct 23 13:24:03 zarafa02 stonith-ng[2259]: notice: stonith_device_register: Added 'stonith-zarafa01' to the device list (1 active devices)
> Oct 23 13:24:04 zarafa02 stonith-ng[2259]: notice: stonith_device_register: Added 'stonith-zarafa02' to the device list (2 active devices)
> Oct 23 13:24:04 zarafa02 crmd[2263]: notice: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
>
>
>
> 2013/10/23 Michael Schwartzkopff <ms at sys4.de>
> Am Mittwoch, 23. Oktober 2013, 12:39:35 schrieb Beo Banks:
> > hi,
> >
> > thanks for answer.
> >
> > the pacemaker|corosync is running on both nodes.
> >
> > [chkconfig | grep corosync
> > corosync 0:Aus 1:Aus 2:Ein 3:Ein 4:Ein 5:Ein 6:Aus
> > chkconfig | grep pacemaker
> > pacemaker 0:Aus 1:Aus 2:Ein 3:Ein 4:Ein 5:Ein 6:Aus
>
> Looks good. What does the log of the boot say? Any signs of corosync?
>
> > @ssh key
> > no, i created the keys without passphrase.
> > maybe the config is wrong but i have checked a lot times and i can´t find
> > any issue.
>
> Is the key readable for the user that executes the fencing command? host list
> correct? How does the fencing agent know to fence "host2" (see your command
> line). How does it know what IP address to use? Is your /etc/hosts correct?
>
> > btw. your book "clusterbau: hochverfügbarkeit mit linux version3" is very
> > helpfull.
>
> Thanks. If your like it, you could write a 5-star review at amazon!
>
> > btw. selinux is disabled and iptables can´t be the reason because
> > fence_virsh works via commandline,or?
>
> it uses the virsh communication. It depends how you did set it up, but
> normally you use SSH. But see your setup.
>
> Mit freundlichen Grüßen,
>
> Michael Schwartzkopff
>
> --
> [*] sys4 AG
>
> http://sys4.de, +49 (89) 30 90 46 64, +49 (162) 165 0044
> Franziskanerstraße 15, 81669 München
>
> Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
> Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
> Aufsichtsratsvorsitzender: Florian Kirstein
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
More information about the Pacemaker
mailing list