[Pacemaker] Migrations Fails to stop iSCSI Target
Terry Miller
tmiller at tmsolutionsgroup.com
Fri Dec 20 06:03:34 UTC 2013
I'm trying to setup a 2-node HA iSCSI cluster but I am having some problems
with the failover. I want each node to be active with half of the storage
available in each node. So each node will have half the storage and it will
be served by different IP/target/lun. I have it working until I have an
initiator setup and test failover. If I have an initiator (proxmox) that is
using the target/lun, then when I migrate lun/target/drbd to the other node,
I get inconsistent results.
Sometimes it works fine, but it usually fails. It seems to be that it fails
in stopping the target after stopping the LUN. At that point the node is
fenced (rebooted). After fencing, sometimes the resources will migrate to
the surviving node successfully, sometimes it will wait until the fenced
node comes back online and then move the resources, and sometimes it will
just remain in the failed state. Sometimes, after the fenced node boots
back up, it will try and move the resources back to itself and this will
fail and the two nodes will continue to fence each other until there is some
manual intervention.
Any assistance if getting this setup correctly would be appreciated.
Relevant details below.
Setup would look like:
On Storage1001A
Lun:vol01 - Target:data01 - IP1/IP11 - LV:lv_vol01 - VG:vg_data01 -
DRBD:data01
On Storage1001B
Lun:vol01 - Target:data02 - IP2/IP22 - LV:lv_vol01 - VG:vg_data02 -
DRBD:data02
Versions:
Linux Storage1001A.xdomain.com 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
GNU/Linux
targetcli 2.0rc1-2
lio-utils 3.1+git2.fd0b34fd-2
drbd8-utils 2:8.3.13-2
corosync 1.4.2-3
pacemaker 1.1.7-1
Network is setup with 3 bonded pairs
- Bond0: Client interface
- Bond1: Crossover
- Bond2: Client interface (for future multicast)
Log file shows
- Working fine
- Migrate from Storage1001A to Storage1001B
- Storage1001A hangs/crashes after stoping LUN before stoppin Target
- Storage1001A is fenced
- Resources migrate to Storage1001B (not 100% sure on this)
- Storage1001A boots back up and tries to take back resources
- Storage1001B hangs after stoping LUN before stoppin Target
- Manually cycle power on Storage1001B
- Resources remain "stuck" on Storage1001B until Storage1001B is back online
(stonith currently disabled for Storage1001B, but sometimes it remains
"stuck" when enabled).
************************************************************
*** Normal Status
************************************************************
# crm status
============
Last updated: Sat Dec 14 14:58:04 2013
Last change: Sat Dec 14 14:45:43 2013 via crm_resource on
Storage1001B.xdomain.com
Stack: openais
Current DC: Storage1001A.xdomain.com - partition with quorum
Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
2 Nodes configured, 2 expected votes
22 Resources configured.
============
Online: [ Storage1001B.xdomain.com Storage1001A.xdomain.com ]
st_node_a (stonith:external/synaccess): Started
Storage1001B.xdomain.com
Resource Group: rg_data01
p_lvm01 (ocf::heartbeat:LVM): Started Storage1001A.xdomain.com
p_ip1 (ocf::heartbeat:IPaddr): Started
Storage1001A.xdomain.com
p_ip11 (ocf::heartbeat:IPaddr): Started
Storage1001A.xdomain.com
p_target_data01 (ocf::heartbeat:iSCSITarget): Started
Storage1001A.xdomain.com
p_lu_data01_vol01 (ocf::heartbeat:iSCSILogicalUnit): Started
Storage1001A.xdomain.com
p_email_admin1 (ocf::heartbeat:MailTo): Started
Storage1001A.xdomain.com
Master/Slave Set: ms_drbd1 [p_drbd1]
Masters: [ Storage1001A.xdomain.com ]
Slaves: [ Storage1001B.xdomain.com ]
Master/Slave Set: ms_drbd2 [p_drbd2]
Masters: [ Storage1001B.xdomain.com ]
Slaves: [ Storage1001A.xdomain.com ]
Clone Set: c_lsb_target [p_target]
Started: [ Storage1001A.xdomain.com Storage1001B.xdomain.com ]
Clone Set: c_ping [p_ping]
Started: [ Storage1001B.xdomain.com Storage1001A.xdomain.com ]
************************************************************
*** Status after migrate
************************************************************
# crm resource migrate rg_data01 Storage1001B.xdomain.com
# crm status
============
Last updated: Sat Dec 14 16:40:55 2013
Last change: Sat Dec 14 16:40:48 2013 via crm_resource on
Storage1001B.xdomain.com
Stack: openais
Current DC: Storage1001A.xdomain.com - partition with quorum
Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
2 Nodes configured, 2 expected votes
22 Resources configured.
============
Online: [ Storage1001B.xdomain.com Storage1001A.xdomain.com ]
st_node_a (stonith:external/synaccess): Started
Storage1001B.xdomain.com
Resource Group: rg_data01
p_lvm01 (ocf::heartbeat:LVM): Started Storage1001A.xdomain.com
p_ip1 (ocf::heartbeat:IPaddr): Started
Storage1001A.xdomain.com
p_ip11 (ocf::heartbeat:IPaddr): Started
Storage1001A.xdomain.com
p_target_data01 (ocf::heartbeat:iSCSITarget): Started
Storage1001A.xdomain.com
p_lu_data01_vol01 (ocf::heartbeat:iSCSILogicalUnit): Stopped
p_email_admin1 (ocf::heartbeat:MailTo): Stopped
Master/Slave Set: ms_drbd1 [p_drbd1]
Masters: [ Storage1001A.xdomain.com ]
Slaves: [ Storage1001B.xdomain.com ]
Master/Slave Set: ms_drbd2 [p_drbd2]
Masters: [ Storage1001B.xdomain.com ]
Slaves: [ Storage1001A.xdomain.com ]
Clone Set: c_lsb_target [p_target]
Started: [ Storage1001A.xdomain.com Storage1001B.xdomain.com ]
Clone Set: c_ping [p_ping]
Started: [ Storage1001B.xdomain.com Storage1001A.xdomain.com ]
************************************************************
*** Status after first node boots back up and tries to take back resource
************************************************************
# crm status
============
Last updated: Tue Dec 17 14:06:15 2013
Last change: Tue Dec 17 14:05:44 2013 via cibadmin on
Storage1001A.xdomain.com
Stack: openais
Current DC: Storage1001B.xdomain.com - partition with quorum
Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
2 Nodes configured, 2 expected votes
22 Resources configured.
============
Node Storage1001B.xdomain.com: UNCLEAN (online)
Online: [ Storage1001A.xdomain.com ]
st_node_a (stonith:external/synaccess): Started
Storage1001B.xdomain.com
Resource Group: rg_data01
p_lvm01 (ocf::heartbeat:LVM): Started Storage1001B.xdomain.com
p_ip1 (ocf::heartbeat:IPaddr): Started
Storage1001B.xdomain.com
p_ip11 (ocf::heartbeat:IPaddr): Started
Storage1001B.xdomain.com
p_target_data01 (ocf::heartbeat:iSCSITarget): Started
Storage1001B.xdomain.com FAILED
p_lu_data01_vol01 (ocf::heartbeat:iSCSILogicalUnit): Stopped
p_email_admin1 (ocf::heartbeat:MailTo): Stopped
Master/Slave Set: ms_drbd1 [p_drbd1]
Masters: [ Storage1001B.xdomain.com ]
Slaves: [ Storage1001A.xdomain.com ]
Master/Slave Set: ms_drbd2 [p_drbd2]
Masters: [ Storage1001B.xdomain.com ]
Slaves: [ Storage1001A.xdomain.com ]
Clone Set: c_lsb_target [p_target]
Started: [ Storage1001A.xdomain.com Storage1001B.xdomain.com ]
Clone Set: c_ping [p_ping]
Started: [ Storage1001A.xdomain.com Storage1001B.xdomain.com ]
Failed actions:
p_target_data01_stop_0 (node=Storage1001B.xdomain.com, call=111, rc=-2,
status=Timed Out): unknown exec error
************************************************************
*** Configuration
*** Note: Some resources are stopped to try and get one
*** resource group working properly
************************************************************
node Storage1001A.xdomain.com
node Storage1001B.xdomain.com
primitive p_drbd1 ocf:linbit:drbd \
params drbd_resource="Data01" \
op monitor interval="3" role="Master" timeout="9" \
op monitor interval="4" role="Slave" timeout="12"
primitive p_drbd2 ocf:linbit:drbd \
params drbd_resource="Data02" \
op monitor interval="3" role="Master" timeout="9" \
op monitor interval="4" role="Slave" timeout="12"
primitive p_email_admin1 ocf:heartbeat:MailTo \
params email="admin at xdomain.com" subject="Cluster Failover"
primitive p_email_admin2 ocf:heartbeat:MailTo \
params email="admin at xdomain.com" subject="Cluster Failover"
primitive p_ip1 ocf:heartbeat:IPaddr \
params ip="10.11.2.13" nic="bond0" cidr_netmask="21" \
op monitor interval="5s"
primitive p_ip11 ocf:heartbeat:IPaddr \
params ip="10.11.10.13" nic="bond2" cidr_netmask="21" \
op monitor interval="5s"
primitive p_ip2 ocf:heartbeat:IPaddr \
params ip="10.11.2.14" nic="bond0" cidr_netmask="21" \
op monitor interval="5s"
primitive p_ip22 ocf:heartbeat:IPaddr \
params ip="10.11.10.14" nic="bond2" cidr_netmask="21" \
op monitor interval="5s"
primitive p_lu_data01_vol01 ocf:heartbeat:iSCSILogicalUnit \
params target_iqn="iqn.2013-10.com.xdomain.storage1001.data01"
lun="1" path="/dev/vg_data01/lv_vol01" implementation="lio" \
op monitor interval="10"
primitive p_lu_data02_vol01 ocf:heartbeat:iSCSILogicalUnit \
params target_iqn="iqn.2013-10.com.xdomain.storage1001.data02"
lun="1" path="/dev/vg_data02/lv_vol01" implementation="lio" \
op monitor interval="10"
primitive p_lvm01 ocf:heartbeat:LVM \
params volgrpname="vg_data01" \
op monitor interval="4" timeout="8"
primitive p_lvm02 ocf:heartbeat:LVM \
params volgrpname="vg_data02" \
op monitor interval="4" timeout="8"
primitive p_ping ocf:pacemaker:ping \
op monitor interval="5s" timeout="15s" \
params host_list="10.11.10.1" multiplier="200" name="p_ping"
primitive p_target lsb:target \
op monitor interval="30" timeout="30"
primitive p_target_data01 ocf:heartbeat:iSCSITarget \
params iqn="iqn.2013-10.com.xdomain.storage1001.data01"
implementation="lio" \
op monitor interval="10s" timeout="20s"
primitive p_target_data02 ocf:heartbeat:iSCSITarget \
params iqn="iqn.2013-10.com.xdomain.storage1001.data02"
implementation="lio" \
op monitor interval="10s" timeout="20s"
primitive st_node_a stonith:external/synaccess \
params synaccessip="reboot11.xdomain.com" community="*******"
port="Storage1001A" pcmk_host_list="Storage1001A.xdomain.com" \
meta target-role="Started"
primitive st_node_b stonith:external/synaccess \
params synaccessip="reboot10.xdomain.com" community="******"
port="Storage1001B" pcmk_host_list="Storage1001B.xdomain.com" \
meta target-role="Stopped"
group rg_data01 p_lvm01 p_ip1 p_ip11 p_target_data01 p_lu_data01_vol01
p_email_admin1 \
meta target-role="Started"
group rg_data02 p_lvm02 p_ip2 p_ip22 p_target_data02 p_lu_data02_vol01
p_email_admin2 \
meta target-role="Stopped"
ms ms_drbd1 p_drbd1 \
meta notify="true" master-max="1" clone-max="2" clone-node-max="1"
target-role="Started" \
meta resource-stickiness="101"
ms ms_drbd2 p_drbd2 \
meta notify="true" master-max="1" clone-max="2" clone-node-max="1"
target-role="Started" \
meta resource-stickiness="101"
clone c_lsb_target p_target \
meta target-role="Started"
clone c_ping p_ping \
meta globally-unique="false" target-role="Started"
location data01_prefer_a ms_drbd1 \
rule $id="data01_prefer_a_rule" $role="Master" 100: #uname eq
Storage1001A.xdomain.com
location data02_prefer_b ms_drbd2 \
rule $id="data02_prefer_b_rule" $role="Master" 100: #uname eq
Storage1001B.xdomain.com
location st_node_a-loc st_node_a \
rule $id="st_node_a-loc-id" -inf: #uname eq Storage1001A.xdomain.com
location st_node_b-loc st_node_b \
rule $id="st_node_b-loc-id" -inf: #uname eq Storage1001B.xdomain.com
colocation c_drbd1 inf: rg_data01 ms_drbd1:Master
colocation c_drbd2 inf: rg_data02 ms_drbd2:Master
order o_data01_start inf: ms_drbd1:promote rg_data01:start
order o_data01_stop inf: rg_data01:stop ms_drbd1:demote
order o_data02_start inf: ms_drbd2:promote rg_data02:start
order o_data02_stop inf: rg_data02:stop ms_drbd2:demote
order o_target_before_data01 inf: c_lsb_target:start rg_data01
order o_target_before_data02 inf: c_lsb_target:start rg_data02
property $id="cib-bootstrap-options" \
dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="true" \
no-quorum-policy="ignore" \
default-resource-stickiness="1" \
last-lrm-refresh="1387064240"
************************************************************
*** Syslog Storage1001A
***********************************************************
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib_process_request:
Operation complete: op cib_delete for section constraints
(origin=local/crm_resource/3, version=0.260.23): ok (rc=0)
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <cib
admin_epoch="0" epoch="260" num_updates="23" >
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <configuration
>
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <constraints
>
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: -
<rsc_location id="cli-prefer-rg_data01" >
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <rule
id="cli-prefer-rule-rg_data01" >
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: -
<expression value="Storage1001A.xdomain.com" id="cli-prefer-expr-rg_data01"
/>
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - </rule>
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: -
</rsc_location>
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: -
</constraints>
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: -
</configuration>
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - </cib>
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: + <cib epoch="261"
num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2"
crm_feature_set="3.0.6" update-origin="Storage1001A.xdomain.com"
update-client="crm_resource" cib-last-written="Tue Dec 17 13:55:53 2013"
have-quorum="1" dc-uuid="Storage1001A.xdomain.com" >
Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: + <configuration
>
Dec 17 14Dec 17 14:05:31 Storage1001A kernel: imklog 5.8.11, log source =
/proc/kmsg started.
Dec 17 14:05:31 Storage1001A rsyslogd: [origin software="rsyslogd"
swVersion="5.8.11" x-pid="2318" x-info="http://www.rsyslog.com"] start
Dec 17 14:05:31 Storage1001A kernel: [ 0.000000] Initializing cgroup
subsys cpuset
Dec 17 14:05:31 Storage1001A kernel: [ 0.000000] Initializing cgroup
subsys cpu
Dec 17 14:05:31 Storage1001A kernel: [ 0.000000] Linux version
3.2.0-4-amd64 (debian-kernel at lists.debian.org) (gcc version 4.6.3 (Debian
4.6.3-14) ) #1 SMP Debian 3.2.51-1
*** Boot Messages Removed ***
Dec 17 14:05:31 Storage1001A kernel: [ 10.404711] fbcon: radeondrmfb (fb0)
is primary device
Dec 17 14:05:31 Storage1001A kernel: [ 10.566191] Console: switching to
colour frame buffer device 128x48
Dec 17 14:05:31 Storage1001A kernel: [ 10.573706] fb0: radeondrmfb frame
buffer device
Dec 17 14:05:31 Storage1001A kernel: [ 10.573707] drm: registered panic
notifier
Dec 17 14:05:31 Storage1001A kernel: [ 10.573713] [drm] Initialized radeon
2.16.0 20080528 for 0000:0c:01.0 on minor 0
Dec 17 14:05:31 Storage1001A kernel: [ 11.841717] EXT4-fs (dm-0):
re-mounted. Opts: (null)
Dec 17 14:05:31 Storage1001A kernel: [ 11.980402] EXT4-fs (dm-0):
re-mounted. Opts: errors=remount-ro
Dec 17 14:05:31 Storage1001A kernel: [ 12.023447] loop: module loaded
Dec 17 14:05:31 Storage1001A kernel: [ 12.324689] Adding 667644k swap on
/dev/mapper/Storage1001A-swap_1. Priority:-1 extents:1 across:667644k
Dec 17 14:05:31 Storage1001A kernel: [ 14.407466] bonding: Ethernet
Channel Bonding Driver: v3.7.1 (April 27, 2011)
Dec 17 14:05:31 Storage1001A kernel: [ 14.407472] bonding: Warning: miimon
must be specified, otherwise bonding will not detect link failure and link
speed which are essential for TLB/ALB load balancing
Dec 17 14:05:31 Storage1001A kernel: [ 14.407475] bonding: Forcing miimon
to 100msec
Dec 17 14:05:31 Storage1001A kernel: [ 14.407477] bonding: In ALB mode you
might experience client disconnections upon reconnection of a link if the
bonding module updelay parameter (0 msec) is incompatible with the
forwarding delay time of the switch
Dec 17 14:05:31 Storage1001A kernel: [ 14.407480] bonding: MII link
monitoring set to 100 ms
Dec 17 14:05:31 Storage1001A kernel: [ 14.410588] bonding: bond0: Setting
MII monitoring interval to 100.
Dec 17 14:05:31 Storage1001A kernel: [ 14.411752] bonding: bond0: setting
mode to balance-alb (6).
Dec 17 14:05:31 Storage1001A kernel: [ 14.413901] bonding: bond0: Adding
slave eth0.
Dec 17 14:05:31 Storage1001A kernel: [ 14.560030] e1000e 0000:05:00.0: irq
67 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 14.617297] e1000e 0000:05:00.0: irq
67 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 14.620289] bonding: bond0:
enslaving eth0 as an active interface with a down link.
Dec 17 14:05:31 Storage1001A kernel: [ 14.623045] bonding: bond0: Adding
slave eth4.
Dec 17 14:05:31 Storage1001A kernel: [ 14.649821] e1000e 0000:07:00.0: irq
71 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 14.705298] e1000e 0000:07:00.0: irq
71 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 14.708389] bonding: bond0:
enslaving eth4 as an active interface with a down link.
Dec 17 14:05:31 Storage1001A kernel: [ 14.712623] e1000e 0000:05:00.0:
eth0: changing MTU from 1500 to 8192
Dec 17 14:05:31 Storage1001A kernel: [ 14.903885] e1000e 0000:07:00.0:
eth4: changing MTU from 1500 to 8192
Dec 17 14:05:31 Storage1001A kernel: [ 14.980307] ADDRCONF(NETDEV_UP):
bond0: link is not ready
Dec 17 14:05:31 Storage1001A kernel: [ 14.997627] bonding: bond1 is being
created...
Dec 17 14:05:31 Storage1001A kernel: [ 14.998786] bonding: bond1: Setting
MII monitoring interval to 100.
Dec 17 14:05:31 Storage1001A kernel: [ 15.001494] bonding: bond1: setting
mode to balance-rr (0).
Dec 17 14:05:31 Storage1001A kernel: [ 15.003576] bonding: bond1: Adding
slave eth1.
Dec 17 14:05:31 Storage1001A kernel: [ 15.151980] e1000e 0000:05:00.1: irq
68 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 15.209292] e1000e 0000:05:00.1: irq
68 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 15.211322] bonding: bond1:
enslaving eth1 as an active interface with a down link.
Dec 17 14:05:31 Storage1001A kernel: [ 15.214034] bonding: bond1: Adding
slave eth5.
Dec 17 14:05:31 Storage1001A kernel: [ 15.249840] e1000e 0000:07:00.1: irq
72 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 15.305290] e1000e 0000:07:00.1: irq
72 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 15.307254] bonding: bond1:
enslaving eth5 as an active interface with a down link.
Dec 17 14:05:31 Storage1001A kernel: [ 15.311595] e1000e 0000:05:00.1:
eth1: changing MTU from 1500 to 9000
Dec 17 14:05:31 Storage1001A kernel: [ 15.499927] e1000e 0000:07:00.1:
eth5: changing MTU from 1500 to 9000
Dec 17 14:05:31 Storage1001A kernel: [ 15.576560] ADDRCONF(NETDEV_UP):
bond1: link is not ready
Dec 17 14:05:31 Storage1001A kernel: [ 15.592865] bonding: bond2 is being
created...
Dec 17 14:05:31 Storage1001A kernel: [ 15.593997] bonding: bond2: Setting
MII monitoring interval to 100.
Dec 17 14:05:31 Storage1001A kernel: [ 15.595266] bonding: bond2: setting
mode to balance-alb (6).
Dec 17 14:05:31 Storage1001A kernel: [ 15.597568] bonding: bond2: Adding
slave eth2.
Dec 17 14:05:31 Storage1001A kernel: [ 15.743998] e1000e 0000:06:00.0: irq
69 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 15.801292] e1000e 0000:06:00.0: irq
69 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 15.804286] bonding: bond2:
enslaving eth2 as an active interface with a down link.
Dec 17 14:05:31 Storage1001A kernel: [ 15.807068] bonding: bond2: Adding
slave eth3.
Dec 17 14:05:31 Storage1001A kernel: [ 15.953819] e1000e 0000:06:00.1: irq
70 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 16.009294] e1000e 0000:06:00.1: irq
70 for MSI/MSI-X
Dec 17 14:05:31 Storage1001A kernel: [ 16.012279] bonding: bond2:
enslaving eth3 as an active interface with a down link.
Dec 17 14:05:31 Storage1001A kernel: [ 16.016311] e1000e 0000:06:00.0:
eth2: changing MTU from 1500 to 8192
Dec 17 14:05:31 Storage1001A kernel: [ 16.207847] e1000e 0000:06:00.1:
eth3: changing MTU from 1500 to 8192
Dec 17 14:05:31 Storage1001A kernel: [ 16.396867] ADDRCONF(NETDEV_UP):
bond2: link is not ready
Dec 17 14:05:31 Storage1001A kernel: [ 16.762069] RPC: Registered named
UNIX socket transport module.
Dec 17 14:05:31 Storage1001A kernel: [ 16.762073] RPC: Registered udp
transport module.
Dec 17 14:05:31 Storage1001A kernel: [ 16.762075] RPC: Registered tcp
transport module.
Dec 17 14:05:31 Storage1001A kernel: [ 16.762076] RPC: Registered tcp
NFSv4.1 backchannel transport module.
Dec 17 14:05:31 Storage1001A kernel: [ 16.782584] FS-Cache: Loaded
Dec 17 14:05:31 Storage1001A kernel: [ 16.827809] FS-Cache: Netfs 'nfs'
registered for caching
Dec 17 14:05:31 Storage1001A kernel: [ 16.851676] Installing knfsd
(copyright (C) 1996 okir at monad.swb.de).
Dec 17 14:05:31 Storage1001A kernel: [ 17.244926] e1000e: eth1 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:05:31 Storage1001A kernel: [ 17.276019] bonding: bond1: link
status definitely up for interface eth1, 1000 Mbps full duplex.
Dec 17 14:05:31 Storage1001A kernel: [ 17.277008] ADDRCONF(NETDEV_CHANGE):
bond1: link becomes ready
Dec 17 14:05:31 Storage1001A kernel: [ 17.288927] e1000e: eth0 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:05:31 Storage1001A kernel: [ 17.376024] bonding: bond0: link
status definitely up for interface eth0, 1000 Mbps full duplex.
Dec 17 14:05:31 Storage1001A kernel: [ 17.376029] bonding: bond0: making
interface eth0 the new active one.
Dec 17 14:05:31 Storage1001A kernel: [ 17.378190] bonding: bond0: first
active interface up!
Dec 17 14:05:31 Storage1001A kernel: [ 17.379114] ADDRCONF(NETDEV_CHANGE):
bond0: link becomes ready
Dec 17 14:05:31 Storage1001A acpid: starting up with netlink and the input
layer
Dec 17 14:05:31 Storage1001A acpid: 1 rule loaded
Dec 17 14:05:31 Storage1001A acpid: waiting for events: event logging is off
Dec 17 14:05:31 Storage1001A kernel: [ 18.003326] e1000e: eth4 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:05:32 Storage1001A kernel: [ 18.076015] bonding: bond0: link
status definitely up for interface eth4, 1000 Mbps full duplex.
Dec 17 14:05:32 Storage1001A logd: [2538]: info: logd started with
/etc/logd.cf.
Dec 17 14:05:32 Storage1001A logd: [2538]: WARN: Core dumps could be lost if
multiple dumps occur.
Dec 17 14:05:32 Storage1001A logd: [2538]: WARN: Consider setting
non-default value in /proc/sys/kernel/core_pattern (or equivalent) for
maximum supportability
Dec 17 14:05:32 Storage1001A logd: [2538]: WARN: Consider setting
/proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum
supportability
Dec 17 14:05:32 Storage1001A kernel: [ 18.388930] e1000e: eth2 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:05:32 Storage1001A kernel: [ 18.392017] bonding: bond2: link
status definitely up for interface eth2, 1000 Mbps full duplex.
Dec 17 14:05:32 Storage1001A kernel: [ 18.392036] bonding: bond2: making
interface eth2 the new active one.
Dec 17 14:05:32 Storage1001A kernel: [ 18.394197] bonding: bond2: first
active interface up!
Dec 17 14:05:32 Storage1001A kernel: [ 18.395186] ADDRCONF(NETDEV_CHANGE):
bond2: link becomes ready
Dec 17 14:05:32 Storage1001A openhpid: ERROR: (init.c, 76, OpenHPI is not
configured. See openhpi.conf file.)
Dec 17 14:05:32 Storage1001A openhpid: ERROR: (openhpid.cpp, 270, There was
an error initializing OpenHPI)
Dec 17 14:05:32 Storage1001A kernel: [ 18.491379] e1000e: eth5 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:05:32 Storage1001A kernel: [ 18.568922] e1000e: eth3 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:05:32 Storage1001A kernel: [ 18.576018] bonding: bond1: link
status definitely up for interface eth5, 1000 Mbps full duplex.
Dec 17 14:05:32 Storage1001A kernel: [ 18.592012] bonding: bond2: link
status definitely up for interface eth3, 1000 Mbps full duplex.
Dec 17 14:05:33 Storage1001A /usr/sbin/cron[2618]: (CRON) INFO (pidfile fd =
3)
Dec 17 14:05:33 Storage1001A /usr/sbin/cron[2626]: (CRON) STARTUP (fork ok)
Dec 17 14:05:33 Storage1001A ntpd[2624]: ntpd 4.2.6p5 at 1.2349-o Sat May 12
09:54:55 UTC 2012 (1)
Dec 17 14:05:33 Storage1001A ntpd[2644]: proto: precision = 0.117 usec
Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen and drop on 0 v4wildcard
0.0.0.0 UDP 123
Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen and drop on 1 v6wildcard ::
UDP 123
Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 2 lo 127.0.0.1
UDP 123
Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 3 bond0
10.11.2.11 UDP 123
Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 4 bond1
10.100.100.11 UDP 123
Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 5 bond2
10.11.10.11 UDP 123
Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 6 bond1
fe80::226:55ff:fed7:3d90 UDP 123
Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 7 lo ::1 UDP 123
Dec 17 14:05:33 Storage1001A ntpd[2644]: peers refreshed
Dec 17 14:05:33 Storage1001A ntpd[2644]: Listening on routing socket on fd
#24 for interface updates
Dec 17 14:05:36 Storage1001A /usr/sbin/cron[2626]: (CRON) INFO (Running
@reboot jobs)
Dec 17 14:05:38 Storage1001A ntpd[2644]: Listen normally on 8 bond2
fe80::226:55ff:fed7:3d93 UDP 123
Dec 17 14:05:38 Storage1001A ntpd[2644]: Listen normally on 9 bond0
fe80::226:55ff:fed7:3d91 UDP 123
Dec 17 14:05:38 Storage1001A ntpd[2644]: peers refreshed
Dec 17 14:05:39 Storage1001A kernel: [ 25.264486] Unable to load
target_core_stgt
Dec 17 14:05:39 Storage1001A kernel: [ 25.562625] drbd: initialized.
Version: 8.3.11 (api:88/proto:86-96)
Dec 17 14:05:39 Storage1001A kernel: [ 25.562628] drbd: srcversion:
F937DCB2E5D83C6CCE4A6C9
Dec 17 14:05:39 Storage1001A kernel: [ 25.562630] drbd: registered as
block device major 147
Dec 17 14:05:39 Storage1001A kernel: [ 25.562632] drbd: minor_table @
0xffff8802249fbdc0
Dec 17 14:05:39 Storage1001A kernel: [ 25.584528] block drbd1: Starting
worker thread (from drbdsetup [2974])
Dec 17 14:05:39 Storage1001A kernel: [ 25.584630] block drbd1: disk(
Diskless -> Attaching )
Dec 17 14:05:39 Storage1001A kernel: [ 25.586424] block drbd1: Found 4
transactions (38 active extents) in activity log.
Dec 17 14:05:39 Storage1001A kernel: [ 25.586427] block drbd1: Method to
ensure write ordering: flush
Dec 17 14:05:39 Storage1001A kernel: [ 25.586429] block drbd1: max BIO
size = 131072
Dec 17 14:05:39 Storage1001A kernel: [ 25.586433] block drbd1:
drbd_bm_resize called with capacity == 3906129416
Dec 17 14:05:39 Storage1001A kernel: [ 25.613463] block drbd1: resync
bitmap: bits=488266177 words=7629160 pages=14901
Dec 17 14:05:39 Storage1001A kernel: [ 25.613469] block drbd1: size = 1863
GB (1953064708 KB)
Dec 17 14:05:40 Storage1001A kernel: [ 26.644370] block drbd1: bitmap READ
of 14901 pages took 258 jiffies
Dec 17 14:05:40 Storage1001A kernel: [ 26.692206] block drbd1: recounting
of set bits took additional 12 jiffies
Dec 17 14:05:40 Storage1001A kernel: [ 26.692211] block drbd1: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Dec 17 14:05:40 Storage1001A kernel: [ 26.692230] block drbd1: Marked
additional 148 MB as out-of-sync based on AL.
Dec 17 14:05:40 Storage1001A kernel: [ 26.692502] block drbd1: bitmap
WRITE of 0 pages took 0 jiffies
Dec 17 14:05:40 Storage1001A kernel: [ 26.693035] block drbd1: 148 MB
(37888 bits) marked out-of-sync by on disk bit-map.
Dec 17 14:05:40 Storage1001A kernel: [ 26.693047] block drbd1: disk(
Attaching -> Consistent )
Dec 17 14:05:40 Storage1001A kernel: [ 26.693051] block drbd1: attached to
UUIDs 82E61159C15A62E0:0000000000000000:711FF2F50EF3A977:711EF2F50EF3A977
Dec 17 14:05:41 Storage1001A kernel: [ 27.472006] bond1: no IPv6 routers
present
Dec 17 14:05:41 Storage1001A kernel: [ 27.704347] block drbd2: Starting
worker thread (from drbdsetup [2990])
Dec 17 14:05:41 Storage1001A kernel: [ 27.704424] block drbd2: disk(
Diskless -> Attaching )
Dec 17 14:05:41 Storage1001A kernel: [ 27.705553] block drbd2: Found 3
transactions (3 active extents) in activity log.
Dec 17 14:05:41 Storage1001A kernel: [ 27.705556] block drbd2: Method to
ensure write ordering: flush
Dec 17 14:05:41 Storage1001A kernel: [ 27.705559] block drbd2: max BIO
size = 131072
Dec 17 14:05:41 Storage1001A kernel: [ 27.705564] block drbd2:
drbd_bm_resize called with capacity == 3906084368
Dec 17 14:05:41 Storage1001A kernel: [ 27.732182] block drbd2: resync
bitmap: bits=488260546 words=7629072 pages=14901
Dec 17 14:05:41 Storage1001A kernel: [ 27.732188] block drbd2: size = 1863
GB (1953042184 KB)
Dec 17 14:05:42 Storage1001A kernel: [ 28.224005] bond0: no IPv6 routers
present
Dec 17 14:05:42 Storage1001A kernel: [ 28.480005] bond2: no IPv6 routers
present
Dec 17 14:05:42 Storage1001A kernel: [ 28.853639] block drbd2: bitmap READ
of 14901 pages took 280 jiffies
Dec 17 14:05:42 Storage1001A kernel: [ 28.901429] block drbd2: recounting
of set bits took additional 12 jiffies
Dec 17 14:05:42 Storage1001A kernel: [ 28.901433] block drbd2: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Dec 17 14:05:42 Storage1001A kernel: [ 28.901444] block drbd2: disk(
Attaching -> Consistent )
Dec 17 14:05:42 Storage1001A kernel: [ 28.901448] block drbd2: attached to
UUIDs 76A87E2130121406:0000000000000000:B007DA0B699331F8:B006DA0B699331F9
Dec 17 14:05:42 Storage1001A kernel: [ 28.928435] block drbd1: conn(
StandAlone -> Unconnected )
Dec 17 14:05:42 Storage1001A kernel: [ 28.928456] block drbd1: Starting
receiver thread (from drbd1_worker [2975])
Dec 17 14:05:42 Storage1001A kernel: [ 28.928514] block drbd1: receiver
(re)started
Dec 17 14:05:42 Storage1001A kernel: [ 28.928526] block drbd1: conn(
Unconnected -> WFConnection )
Dec 17 14:05:42 Storage1001A kernel: [ 28.929753] block drbd2: conn(
StandAlone -> Unconnected )
Dec 17 14:05:42 Storage1001A kernel: [ 28.929762] block drbd2: Starting
receiver thread (from drbd2_worker [2991])
Dec 17 14:05:42 Storage1001A kernel: [ 28.929813] block drbd2: receiver
(re)started
Dec 17 14:05:42 Storage1001A kernel: [ 28.929820] block drbd2: conn(
Unconnected -> WFConnection )
Dec 17 14:05:43 Storage1001A kernel: [ 29.427516] block drbd1: Handshake
successful: Agreed network protocol version 96
Dec 17 14:05:43 Storage1001A kernel: [ 29.427526] block drbd1: conn(
WFConnection -> WFReportParams )
Dec 17 14:05:43 Storage1001A kernel: [ 29.427563] block drbd1: Starting
asender thread (from drbd1_receiver [3003])
Dec 17 14:05:43 Storage1001A kernel: [ 29.427567] block drbd2: Handshake
successful: Agreed network protocol version 96
Dec 17 14:05:43 Storage1001A kernel: [ 29.427574] block drbd2: conn(
WFConnection -> WFReportParams )
Dec 17 14:05:43 Storage1001A kernel: [ 29.427611] block drbd2: Starting
asender thread (from drbd2_receiver [3007])
Dec 17 14:05:43 Storage1001A kernel: [ 29.427718] block drbd2:
data-integrity-alg: <not-used>
Dec 17 14:05:43 Storage1001A kernel: [ 29.427729] block drbd1:
data-integrity-alg: <not-used>
Dec 17 14:05:43 Storage1001A kernel: [ 29.427764] block drbd1:
drbd_sync_handshake:
Dec 17 14:05:43 Storage1001A kernel: [ 29.427768] block drbd1: self
82E61159C15A62E0:0000000000000000:711FF2F50EF3A977:711EF2F50EF3A977
bits:37888 flags:0
Dec 17 14:05:43 Storage1001A kernel: [ 29.427771] block drbd1: peer
E48A0D04194BDA73:82E61159C15A62E0:711FF2F50EF3A976:711EF2F50EF3A977 bits:0
flags:0
Dec 17 14:05:43 Storage1001A kernel: [ 29.427774] block drbd1:
uuid_compare()=-1 by rule 50
Dec 17 14:05:43 Storage1001A kernel: [ 29.427778] block drbd2:
drbd_sync_handshake:
Dec 17 14:05:43 Storage1001A kernel: [ 29.427782] block drbd2: self
76A87E2130121406:0000000000000000:B007DA0B699331F8:B006DA0B699331F9 bits:0
flags:0
Dec 17 14:05:43 Storage1001A kernel: [ 29.427786] block drbd2: peer
2A0715EFC37BB5BB:76A87E2130121407:B007DA0B699331F9:B006DA0B699331F9 bits:0
flags:0
Dec 17 14:05:43 Storage1001A kernel: [ 29.427792] block drbd1: peer(
Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent ->
Outdated ) pdsk( DUnknown -> UpToDate )
Dec 17 14:05:43 Storage1001A kernel: [ 29.427800] block drbd2:
uuid_compare()=-1 by rule 50
Dec 17 14:05:43 Storage1001A kernel: [ 29.427807] block drbd2: peer(
Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent ->
Outdated ) pdsk( DUnknown -> UpToDate )
Dec 17 14:05:43 Storage1001A kernel: [ 30.168469] block drbd1: conn(
WFBitMapT -> WFSyncUUID )
Dec 17 14:05:43 Storage1001A corosync[3057]: [MAIN ] Corosync Cluster
Engine ('1.4.2'): started and ready to provide service.
Dec 17 14:05:43 Storage1001A corosync[3057]: [MAIN ] Corosync built-in
features: nss
Dec 17 14:05:43 Storage1001A corosync[3057]: [MAIN ] Successfully read
main configuration file '/etc/corosync/corosync.conf'.
Dec 17 14:05:43 Storage1001A corosync[3057]: [TOTEM ] Initializing
transport (UDP/IP Unicast).
Dec 17 14:05:43 Storage1001A corosync[3057]: [TOTEM ] Initializing
transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Dec 17 14:05:44 Storage1001A kernel: [ 30.271375] block drbd1: updated
sync uuid
82E71159C15A62E0:0000000000000000:711FF2F50EF3A977:711EF2F50EF3A977
Dec 17 14:05:44 Storage1001A corosync[3057]: [TOTEM ] The network
interface [10.100.100.11] is now up.
Dec 17 14:05:44 Storage1001A kernel: [ 30.294016] block drbd1: helper
command: /sbin/drbdadm before-resync-target minor-1
Dec 17 14:05:44 Storage1001A kernel: [ 30.296565] block drbd1: helper
command: /sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)
Dec 17 14:05:44 Storage1001A kernel: [ 30.296577] block drbd1: conn(
WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
Dec 17 14:05:44 Storage1001A kernel: [ 30.296587] block drbd1: Began
resync as SyncTarget (will sync 151552 KB [37888 bits set]).
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
process_ais_conf: Reading configure
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
config_find_init: Local handle: 2730409743423111171 for logging
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
config_find_next: Processing additional logging options...
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
get_config_opt: Found 'off' for option: debug
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
get_config_opt: Found 'no' for option: to_logfile
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
get_config_opt: Found 'yes' for option: to_syslog
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
get_config_opt: Found 'daemon' for option: syslog_facility
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
config_find_init: Local handle: 5880381755227111428 for quorum
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
config_find_next: Processing additional quorum options...
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
get_config_opt: Found 'corosync_votequorum' for option: provider
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
config_find_init: Local handle: 4835695805891346437 for service
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
config_find_next: Processing additional service options...
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
get_config_opt: Found '0' for option: ver
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
get_config_opt: Defaulting to 'pcmk' for option: clustername
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
get_config_opt: Defaulting to 'no' for option: use_logd
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
get_config_opt: Defaulting to 'no' for option: use_mgmtd
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_startup:
CRM: Initialized
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] Logging: Initialized
pcmk_startup
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_startup:
Maximum core file size is: 18446744073709551615
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_startup:
Service: 9
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_startup:
Local hostname: Storage1001A.xdomain.com
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
pcmk_update_nodeid: Local node id: 191128586
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
Creating entry for node 191128586 born on 0
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
0x1997aa0 Node 191128586 now known as Storage1001A.xdomain.com (was: (null))
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
Node Storage1001A.xdomain.com now has 1 quorum votes (was 0)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
Node 191128586/Storage1001A.xdomain.com is now: member
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child:
Forked child 3095 for process cib
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child:
Forked child 3096 for process stonith-ng
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child:
Forked child 3097 for process lrmd
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child:
Forked child 3098 for process attrd
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child:
Forked child 3099 for process pengine
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child:
Forked child 3100 for process crmd
Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine
loaded: Pacemaker Cluster Manager 1.1.7
Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine
loaded: corosync extended virtual synchrony service
Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine
loaded: corosync configuration service
Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine
loaded: corosync cluster closed process group service v1.01
Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine
loaded: corosync cluster config database access v1.01
Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine
loaded: corosync profile loading service
Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Using quorum
provider corosync_votequorum
Dec 17 14:05:44 Storage1001A corosync[3057]: [VOTEQ ] Reading
configuration
Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[0]:
Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine
loaded: corosync votes quorum service v0.91
Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine
loaded: corosync cluster quorum service v0.1
Dec 17 14:05:44 Storage1001A corosync[3057]: [MAIN ] Compatibility mode
set to whitetank. Using V1 and V2 of the synchronization engine.
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] notice:
pcmk_peer_update: Transitional membership event on ring 1888: memb=0, new=0,
lost=0
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] notice:
pcmk_peer_update: Stable membership event on ring 1888: memb=1, new=1,
lost=0
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
pcmk_peer_update: NEW: Storage1001A.xdomain.com 191128586
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
pcmk_peer_update: MEMB: Storage1001A.xdomain.com 191128586
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
Node Storage1001A.xdomain.com now has process list:
00000000000000000000000000111312 (1118994)
Dec 17 14:05:44 Storage1001A corosync[3057]: [TOTEM ] A processor joined
or left the membership and a new membership was formed.
Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[1]:
191128586
Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[1]:
191128586
Dec 17 14:05:44 Storage1001A corosync[3057]: [CPG ] chosen downlist:
sender r(0) ip(10.100.100.11) ; members(old:0 left:0)
Dec 17 14:05:44 Storage1001A corosync[3057]: [MAIN ] Completed service
synchronization, ready to provide service.
Dec 17 14:05:44 Storage1001A kernel: [ 30.403515] block drbd2: conn(
WFBitMapT -> WFSyncUUID )
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] notice:
pcmk_peer_update: Transitional membership event on ring 1892: memb=1, new=0,
lost=0
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
pcmk_peer_update: memb: Storage1001A.xdomain.com 191128586
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] notice:
pcmk_peer_update: Stable membership event on ring 1892: memb=2, new=1,
lost=0
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
Creating entry for node 207905802 born on 1892
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
Node 207905802/unknown is now: member
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
pcmk_peer_update: NEW: .pending. 207905802
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
pcmk_peer_update: MEMB: Storage1001A.xdomain.com 191128586
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
pcmk_peer_update: MEMB: .pending. 207905802
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
send_member_notification: Sending membership update 1892 to 0 children
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
0x1997aa0 Node 191128586 ((null)) born on: 1892
Dec 17 14:05:44 Storage1001A corosync[3057]: [TOTEM ] A processor joined
or left the membership and a new membership was formed.
Dec 17 14:05:44 Storage1001A corosync[3057]: [VOTEQ ] quorum regained,
resuming activity
Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] This node is within
the primary component and will provide service.
Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[2]:
191128586 207905802
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
0x199f440 Node 207905802 (Storage1001B.xdomain.com) born on: 1884
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
0x199f440 Node 207905802 now known as Storage1001B.xdomain.com (was: (null))
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
Node Storage1001B.xdomain.com now has process list:
00000000000000000000000000111312 (1118994)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member:
Node Storage1001B.xdomain.com now has 1 quorum votes (was 0)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info:
send_member_notification: Sending membership update 1892 to 0 children
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.crmd failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[2]:
191128586 207905802
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.crmd failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A kernel: [ 30.461235] block drbd2: updated
sync uuid
76A97E2130121406:0000000000000000:B007DA0B699331F8:B006DA0B699331F9
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [CPG ] chosen downlist:
sender r(0) ip(10.100.100.11) ; members(old:1 left:0)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [MAIN ] Completed service
synchronization, ready to provide service.
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.crmd failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN:
route_ais_message: Sending message to local.cib failed: ipc delivery failed
(rc=-2)
Dec 17 14:05:44 Storage1001A kernel: [ 30.551166] block drbd2: helper
command: /sbin/drbdadm before-resync-target minor-2
Dec 17 14:05:44 Storage1001A kernel: [ 30.554239] block drbd2: helper
command: /sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Dec 17 14:05:44 Storage1001A kernel: [ 30.554250] block drbd2: conn(
WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
Dec 17 14:05:44 Storage1001A kernel: [ 30.554259] block drbd2: Began
resync as SyncTarget (will sync 0 KB [0 bits set]).
Dec 17 14:05:44 Storage1001A lrmd: [3097]: info: enabling coredumps
Dec 17 14:05:44 Storage1001A lrmd: [3097]: WARN: Core dumps could be lost if
multiple dumps occur.
Dec 17 14:05:44 Storage1001A lrmd: [3097]: WARN: Consider setting
non-default value in /proc/sys/kernel/core_pattern (or equivalent) for
maximum supportability
Dec 17 14:05:44 Storage1001A lrmd: [3097]: WARN: Consider setting
/proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum
supportability
Dec 17 14:05:44 Storage1001A lrmd: [3097]: info: Started.
Dec 17 14:05:44 Storage1001A kernel: [ 30.557373] block drbd2: Resync done
(total 1 sec; paused 0 sec; 0 K/sec)
Dec 17 14:05:44 Storage1001A kernel: [ 30.557379] block drbd2: updated
UUIDs 2A0715EFC37BB5BA:0000000000000000:76A97E2130121406:76A87E2130121407
Dec 17 14:05:44 Storage1001A kernel: [ 30.557384] block drbd2: conn(
SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: Invoked:
/usr/lib/pacemaker/stonithd
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: crm_log_init_worker:
Changed active directory to /var/lib/heartbeat/cores/root
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: get_cluster_type:
Cluster type is: 'openais'
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: notice:
crm_cluster_connect: Connecting to cluster infrastructure: classic openais
(with plugin)
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info:
init_ais_connection_classic: Creating connection to our Corosync plugin
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info:
init_ais_connection_classic: AIS connection established
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc:
Recorded connection 0x19a3a60 for stonith-ng/3096
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: get_ais_nodeid:
Server details: id=191128586 uname=Storage1001A.xdomain.com cname=pcmk
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info:
init_ais_connection_once: Connection to 'classic openais (with plugin)':
established
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: crm_new_peer: Node
Storage1001A.xdomain.com now has id: 191128586
Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: crm_new_peer: Node
191128586 is now known as Storage1001A.xdomain.com
Dec 17 14:05:44 Storage1001A pengine: [3099]: info: Invoked:
/usr/lib/pacemaker/pengine
Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_log_init_worker: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Dec 17 14:05:44 Storage1001A attrd: [3098]: info: Invoked:
/usr/lib/pacemaker/attrd
Dec 17 14:05:44 Storage1001A attrd: [3098]: notice: crm_cluster_connect:
Connecting to cluster infrastructure: classic openais (with plugin)
Dec 17 14:05:44 Storage1001A cib: [3095]: info: retrieveCib: Reading cluster
configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
/var/lib/heartbeat/crm/cib.xml.sig)
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc:
Recorded connection 0x19a7dc0 for attrd/3098
Dec 17 14:05:44 Storage1001A attrd: [3098]: notice: main: Starting
mainloop...
Dec 17 14:05:44 Storage1001A cib: [3095]: info: validate_with_relaxng:
Creating RNG parser context
Dec 17 14:05:44 Storage1001A cib: [3095]: info: startCib: CIB Initialization
completed successfully
Dec 17 14:05:44 Storage1001A cib: [3095]: info: get_cluster_type: Cluster
type is: 'openais'
Dec 17 14:05:44 Storage1001A cib: [3095]: notice: crm_cluster_connect:
Connecting to cluster infrastructure: classic openais (with plugin)
Dec 17 14:05:44 Storage1001A cib: [3095]: info: init_ais_connection_classic:
Creating connection to our Corosync plugin
Dec 17 14:05:44 Storage1001A cib: [3095]: info: init_ais_connection_classic:
AIS connection established
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc:
Recorded connection 0x19ac120 for cib/3095
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc:
Sending membership update 1892 to cib
Dec 17 14:05:44 Storage1001A cib: [3095]: info: get_ais_nodeid: Server
details: id=191128586 uname=Storage1001A.xdomain.com cname=pcmk
Dec 17 14:05:44 Storage1001A cib: [3095]: info: init_ais_connection_once:
Connection to 'classic openais (with plugin)': established
Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_new_peer: Node
Storage1001A.xdomain.com now has id: 191128586
Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_new_peer: Node 191128586
is now known as Storage1001A.xdomain.com
Dec 17 14:05:44 Storage1001A cib: [3095]: info: cib_init: Starting cib
mainloop
Dec 17 14:05:44 Storage1001A cib: [3095]: notice: ais_dispatch_message:
Membership 1892: quorum acquired
Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_update_peer: Node
Storage1001A.xdomain.com: id=191128586 state=member (new) addr=r(0)
ip(10.100.100.11) (new) votes=1 (new) born=1892 seen=1892
proc=00000000000000000000000000111312 (new)
Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_new_peer: Node
Storage1001B.xdomain.com now has id: 207905802
Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_new_peer: Node 207905802
is now known as Storage1001B.xdomain.com
Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_update_peer: Node
Storage1001B.xdomain.com: id=207905802 state=member (new) addr=r(0)
ip(10.100.100.12) votes=1 born=1884 seen=1892
proc=00000000000000000000000000111312
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: Invoked:
/usr/lib/pacemaker/crmd
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_log_init_worker:
Changed active directory to /var/lib/heartbeat/cores/hacluster
Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: main: CRM Git Version:
ee0730e13d124c3d58f00016c3376a1de5323cff
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: do_cib_control: CIB
connection established
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: get_cluster_type: Cluster
type is: 'openais'
Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: crm_cluster_connect:
Connecting to cluster infrastructure: classic openais (with plugin)
Dec 17 14:05:44 Storage1001A crmd: [3100]: info:
init_ais_connection_classic: Creating connection to our Corosync plugin
Dec 17 14:05:44 Storage1001A crmd: [3100]: info:
init_ais_connection_classic: AIS connection established
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc:
Recorded connection 0x19b06c0 for crmd/3100
Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc:
Sending membership update 1892 to crmd
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: get_ais_nodeid: Server
details: id=191128586 uname=Storage1001A.xdomain.com cname=pcmk
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: init_ais_connection_once:
Connection to 'classic openais (with plugin)': established
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_new_peer: Node
Storage1001A.xdomain.com now has id: 191128586
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_new_peer: Node
191128586 is now known as Storage1001A.xdomain.com
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: ais_status_callback:
status: Storage1001A.xdomain.com is now unknown
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: do_ha_control: Connected to
the cluster
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: do_started: Delaying start,
no membership data (0000000000100000)
Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: ais_dispatch_message:
Membership 1892: quorum acquired
Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: crmd_peer_update: Status
update: Client Storage1001A.xdomain.com/crmd now has status [online]
(DC=<null>)
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: ais_status_callback:
status: Storage1001A.xdomain.com is now member (was unknown)
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_update_peer: Node
Storage1001A.xdomain.com: id=191128586 state=member (new) addr=r(0)
ip(10.100.100.11) (new) votes=1 (new) born=1892 seen=1892
proc=00000000000000000000000000111312 (new)
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_new_peer: Node
Storage1001B.xdomain.com now has id: 207905802
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_new_peer: Node
207905802 is now known as Storage1001B.xdomain.com
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: ais_status_callback:
status: Storage1001B.xdomain.com is now unknown
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: ais_status_callback:
status: Storage1001B.xdomain.com is now member (was unknown)
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_update_peer: Node
Storage1001B.xdomain.com: id=207905802 state=member (new) addr=r(0)
ip(10.100.100.12) votes=1 born=1884 seen=1892
proc=00000000000000000000000000111312
Dec 17 14:05:44 Storage1001A crmd: [3100]: info: do_started: Delaying start,
Config not read (0000000000000040)
Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: do_started: The local CRM
is operational
Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: do_state_transition:
State transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_started ]
Dec 17 14:05:44 Storage1001A kernel: [ 30.776140] block drbd2: helper
command: /sbin/drbdadm after-resync-target minor-2
Dec 17 14:05:44 Storage1001A crm-unfence-peer.sh[3113]: invoked for Data02
Dec 17 14:05:44 Storage1001A cib: [3095]: info: cib_process_diff: Diff
0.262.25 -> 0.263.1 not applied to 0.261.0: current "epoch" is less than
required
Dec 17 14:05:44 Storage1001A cib: [3095]: info: cib_server_process_diff:
Requesting re-sync from peer
Dec 17 14:05:44 Storage1001A kernel: [ 30.860423] block drbd2: helper
command: /sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Dec 17 14:05:44 Storage1001A cib: [3095]: info: cib_replace_notify:
Replaced: -1.-1.-1 -> 0.263.1 from Storage1001B.xdomain.com
Dec 17 14:05:44 Storage1001A kernel: [ 31.174689] block drbd2: bitmap
WRITE of 14901 pages took 78 jiffies
Dec 17 14:05:45 Storage1001A kernel: [ 31.526981] block drbd2: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Dec 17 14:05:45 Storage1001A stonith-ng: [3096]: notice: setup_cib: Watching
for stonith topology changes
Dec 17 14:05:45 Storage1001A stonith-ng: [3096]: info: main: Starting
stonith-ng mainloop
Dec 17 14:05:45 Storage1001A crmd: [3100]: info: ais_dispatch_message:
Membership 1892: quorum retained
Dec 17 14:05:46 Storage1001A kernel: [ 32.599664] block drbd1: Resync done
(total 2 sec; paused 0 sec; 75776 K/sec)
Dec 17 14:05:46 Storage1001A kernel: [ 32.599671] block drbd1: updated
UUIDs E48A0D04194BDA72:0000000000000000:82E71159C15A62E0:82E61159C15A62E0
Dec 17 14:05:46 Storage1001A kernel: [ 32.599680] block drbd1: conn(
SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
Dec 17 14:05:46 Storage1001A kernel: [ 32.652349] block drbd1: helper
command: /sbin/drbdadm after-resync-target minor-1
Dec 17 14:05:46 Storage1001A crm-unfence-peer.sh[3132]: invoked for Data01
Dec 17 14:05:46 Storage1001A kernel: [ 32.695859] block drbd1: helper
command: /sbin/drbdadm after-resync-target minor-1 exit code 0 (0x0)
Dec 17 14:05:46 Storage1001A crmd: [3100]: info: update_dc: Set DC to
Storage1001B.xdomain.com (3.0.6)
Dec 17 14:05:46 Storage1001A cib: [3095]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=Storage1001B.xdomain.com/crmd/130, version=0.264.1): ok (rc=0)
Dec 17 14:05:46 Storage1001A crmd: [3100]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='Storage1001A.xdomain.com']/transient_attributes
Dec 17 14:05:46 Storage1001A crmd: [3100]: info: update_attrd: Connecting to
attrd...
Dec 17 14:05:46 Storage1001A crmd: [3100]: notice: do_state_transition:
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE
origin=do_cl_join_finalize_respond ]
Dec 17 14:05:46 Storage1001A attrd: [3098]: notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Dec 17 14:05:46 Storage1001A kernel: [ 32.787912] block drbd1: bitmap
WRITE of 14889 pages took 23 jiffies
Dec 17 14:05:46 Storage1001A kernel: [ 32.986722] block drbd1: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Dec 17 14:05:47 Storage1001A lrmd: [3097]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:st_node_a probe[2] (pid
3148)
Dec 17 14:05:47 Storage1001A lrmd: [3097]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:st_node_b probe[3] (pid
3149)
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:p_lvm01 probe[4] (pid
3150)
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:p_ip1 probe[5] (pid
3151)
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:p_ip11 probe[6] (pid
3152)
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:p_target_data01
probe[7] (pid 3153)
Dec 17 14:05:47 Storage1001A lrmd: [3097]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Dec 17 14:05:47 Storage1001A stonith-ng: [3096]: notice:
stonith_device_action: Device st_node_b not found
Dec 17 14:05:47 Storage1001A stonith-ng: [3096]: info: stonith_command:
Processed st_execute from lrmd: rc=-12
Dec 17 14:05:47 Storage1001A stonith-ng: [3096]: notice:
stonith_device_action: Device st_node_a not found
Dec 17 14:05:47 Storage1001A stonith-ng: [3096]: info: stonith_command:
Processed st_execute from lrmd: rc=-12
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[3] on
st_node_b for client 3100: pid 3149 exited with return code 7
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[2] on
st_node_a for client 3100: pid 3148 exited with return code 7
Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation st_node_b_monitor_0 (call=3, rc=7, cib-update=7, confirmed=true)
not running
Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation st_node_a_monitor_0 (call=2, rc=7, cib-update=8, confirmed=true)
not running
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[5] on
p_ip1 for client 3100: pid 3151 exited with return code 7
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[6] on
p_ip11 for client 3100: pid 3152 exited with return code 7
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[7] on
p_target_data01 for client 3100: pid 3153 exited with return code 0
Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_ip1_monitor_0 (call=5, rc=7, cib-update=9, confirmed=true) not
running
Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_ip11_monitor_0 (call=6, rc=7, cib-update=10, confirmed=true) not
running
Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_target_data01_monitor_0 (call=7, rc=0, cib-update=11,
confirmed=true) ok
Dec 17 14:05:47 Storage1001A LVM[3150]: INFO: LVM Volume vg_data01 is
offline
Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[4] on
p_lvm01 for client 3100: pid 3150 exited with return code 7
Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_lvm01_monitor_0 (call=4, rc=7, cib-update=12, confirmed=true)
not running
Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: rsc:p_lu_data01_vol01
probe[8] (pid 3241)
Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: rsc:p_email_admin1 probe[9]
(pid 3242)
Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: rsc:p_lvm02 probe[10] (pid
3243)
Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: rsc:p_ip2 probe[11] (pid
3244)
Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: operation monitor[11] on
p_ip2 for client 3100: pid 3244 exited with return code 7
Dec 17 14:05:48 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_ip2_monitor_0 (call=11, rc=7, cib-update=13, confirmed=true) not
running
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[9] on
p_email_admin1 for client 3100: pid 3242 exited with return code 7
Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_email_admin1_monitor_0 (call=9, rc=7, cib-update=14,
confirmed=true) not running
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[8] on
p_lu_data01_vol01 for client 3100: pid 3241 exited with return code 7
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: rsc:p_ip22 probe[12] (pid
3310)
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: rsc:p_target_data02
probe[13] (pid 3311)
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: rsc:p_lu_data02_vol01
probe[14] (pid 3312)
Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_lu_data01_vol01_monitor_0 (call=8, rc=7, cib-update=15,
confirmed=true) not running
Dec 17 14:05:49 Storage1001A LVM[3243]: INFO: LVM Volume vg_data02 is
offline
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[10] on
p_lvm02 for client 3100: pid 3243 exited with return code 7
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: rsc:p_drbd1:0 probe[16]
(pid 3332)
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[12] on
p_ip22 for client 3100: pid 3310 exited with return code 7
Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_lvm02_monitor_0 (call=10, rc=7, cib-update=16, confirmed=true)
not running
Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_ip22_monitor_0 (call=12, rc=7, cib-update=17, confirmed=true)
not running
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[13] on
p_target_data02 for client 3100: pid 3311 exited with return code 7
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[14] on
p_lu_data02_vol01 for client 3100: pid 3312 exited with return code 7
Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_target_data02_monitor_0 (call=13, rc=7, cib-update=18,
confirmed=true) not running
Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_lu_data02_vol01_monitor_0 (call=14, rc=7, cib-update=19,
confirmed=true) not running
Dec 17 14:05:49 Storage1001A attrd: [3098]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p_drbd1:0 (10000)
Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[16] on
p_drbd1:0 for client 3100: pid 3332 exited with return code 0
Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_drbd1:0_monitor_0 (call=16, rc=0, cib-update=20, confirmed=true)
ok
Dec 17 14:05:49 Storage1001A attrd: [3098]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p_drbd1:0 (10000)
Dec 17 14:05:49 Storage1001A attrd: [3098]: notice: attrd_perform_update:
Sent update 4: master-p_drbd1:0=10000
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_email_admin2
probe[15] (pid 3396)
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: operation monitor[15] on
p_email_admin2 for client 3100: pid 3396 exited with return code 7
Dec 17 14:05:50 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_email_admin2_monitor_0 (call=15, rc=7, cib-update=21,
confirmed=true) not running
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_drbd2:0 probe[17]
(pid 3400)
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_target:0 probe[18]
(pid 3401)
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_ping:0 probe[19] (pid
3402)
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: operation monitor[19] on
p_ping:0 for client 3100: pid 3402 exited with return code 7
Dec 17 14:05:50 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_ping:0_monitor_0 (call=19, rc=7, cib-update=22, confirmed=true)
not running
Dec 17 14:05:50 Storage1001A attrd: [3098]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p_drbd2:0 (10000)
Dec 17 14:05:50 Storage1001A attrd: [3098]: notice: attrd_perform_update:
Sent update 7: master-p_drbd2:0=10000
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: operation monitor[17] on
p_drbd2:0 for client 3100: pid 3400 exited with return code 0
Dec 17 14:05:50 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_drbd2:0_monitor_0 (call=17, rc=0, cib-update=23, confirmed=true)
ok
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: operation monitor[18] on
p_target:0 for client 3100: pid 3401 exited with return code 0
Dec 17 14:05:50 Storage1001A crmd: [3100]: info: process_lrm_event: LRM
operation p_target:0_monitor_0 (call=18, rc=0, cib-update=24,
confirmed=true) ok
Dec 17 14:05:50 Storage1001A attrd: [3098]: notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Dec 17 14:05:50 Storage1001A attrd: [3098]: notice: attrd_perform_update:
Sent update 10: probe_complete=true
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_drbd1:0 monitor[20]
(pid 3460)
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_drbd2:0 monitor[21]
(pid 3461)
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_target:0 monitor[22]
(pid 3462)
Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_ping:0 start[23] (pid
3463)
************************************************************
*** Syslog Storage1001B
***********************************************************
Dec 17 14:03:36 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[85]
(pid 13822)
Dec 17 14:03:36 Storage1001B lrmd: [3070]: info: operation notify[85] on
p_drbd1:1 for client 3073: pid 13822 exited with return code 0
Dec 17 14:03:36 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_notify_0 (call=85, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:03:56 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[86]
(pid 14131)
Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: crm_new_peer: Node
Storage1001A.xdomain.com now has id: 191128586
Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: crm_new_peer: Node
191128586 is now known as Storage1001A.xdomain.com
Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info:
can_fence_host_with_device: st_node_a can fence Storage1001A.xdomain.com:
static-list
Dec 17 14:03:56 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[87]
(pid 14132)
Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info:
can_fence_host_with_device: st_node_a can fence Storage1001A.xdomain.com:
static-list
Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: stonith_fence: Found
1 matching devices for 'Storage1001A.xdomain.com'
Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: stonith_command:
Processed st_fence from Storage1001A.xdomain.com: rc=-1
Dec 17 14:03:56 Storage1001B lrmd: [3070]: info: operation notify[86] on
p_drbd1:1 for client 3073: pid 14131 exited with return code 0
Dec 17 14:03:56 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_notify_0 (call=86, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:03:56 Storage1001B lrmd: [3070]: info: operation notify[87] on
p_drbd2:1 for client 3073: pid 14132 exited with return code 0
Dec 17 14:03:56 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd2:1_notify_0 (call=87, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:03:57 Storage1001B kernel: [ 643.872134] e1000e: eth1 NIC Link is
Down
Dec 17 14:03:57 Storage1001B kernel: [ 643.872233] e1000e: eth5 NIC Link is
Down
Dec 17 14:03:57 Storage1001B kernel: [ 643.936009] bonding: bond1: link
status definitely down for interface eth1, disabling it
Dec 17 14:03:57 Storage1001B kernel: [ 643.936015] bonding: bond1: now
running without any active interface !
Dec 17 14:03:57 Storage1001B kernel: [ 643.936019] bonding: bond1: link
status definitely down for interface eth5, disabling it
Dec 17 14:03:58 Storage1001B cib: [3068]: info: cib_stats: Processed 471
operations (84.00us average, 0% utilization) in the last 10min
Dec 17 14:03:59 Storage1001B kernel: [ 645.488010] block drbd2: PingAck did
not arrive in time.
Dec 17 14:03:59 Storage1001B kernel: [ 645.488106] block drbd2: peer(
Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate ->
DUnknown )
Dec 17 14:03:59 Storage1001B kernel: [ 645.488141] block drbd2: new current
UUID 2A0715EFC37BB5BB:76A87E2130121407:B007DA0B699331F9:B006DA0B699331F9
Dec 17 14:03:59 Storage1001B kernel: [ 645.512675] block drbd2: asender
terminated
Dec 17 14:03:59 Storage1001B kernel: [ 645.512680] block drbd2: Terminating
drbd2_asender
Dec 17 14:03:59 Storage1001B kernel: [ 645.521157] block drbd2: Connection
closed
Dec 17 14:03:59 Storage1001B kernel: [ 645.521203] block drbd2: conn(
NetworkFailure -> Unconnected )
Dec 17 14:03:59 Storage1001B kernel: [ 645.521209] block drbd2: helper
command: /sbin/drbdadm fence-peer minor-2
Dec 17 14:03:59 Storage1001B kernel: [ 645.521216] block drbd2: receiver
terminated
Dec 17 14:03:59 Storage1001B kernel: [ 645.521226] block drbd2: Restarting
drbd2_receiver
Dec 17 14:03:59 Storage1001B kernel: [ 645.521236] block drbd2: receiver
(re)started
Dec 17 14:03:59 Storage1001B kernel: [ 645.521241] block drbd2: conn(
Unconnected -> WFConnection )
Dec 17 14:03:59 Storage1001B crm-fence-peer.sh[14270]: invoked for Data02
Dec 17 14:03:59 Storage1001B crm-fence-peer.sh[14270]: INFO suitable
constraint already placed: 'drbd-fence-by-handler-Data02-ms_drbd2'
Dec 17 14:03:59 Storage1001B kernel: [ 645.555063] block drbd2: helper
command: /sbin/drbdadm fence-peer minor-2 exit code 4 (0x400)
Dec 17 14:03:59 Storage1001B kernel: [ 645.555068] block drbd2: fence-peer
helper returned 4 (peer was fenced)
Dec 17 14:03:59 Storage1001B kernel: [ 645.555076] block drbd2: pdsk(
DUnknown -> Outdated )
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="262"
num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2"
crm_feature_set="3.0.6" update-origin="Storage1001A.xdomain.com"
update-client="crm_resource" cib-last-written="Tue Dec 17 14:03:36 2013"
have-quorum="0" dc-uuid="Storage1001B.xdomain.com" >
Dec 17 14:04:00 Storage1001B corosync[3033]: [TOTEM ] A processor failed,
forming new configuration.
Dec 17 14:04:01 Storage1001B kernel: [ 647.400010] block drbd1: PingAck did
not arrive in time.
Dec 17 14:04:01 Storage1001B kernel: [ 647.400111] block drbd1: peer(
Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate ->
DUnknown )
Dec 17 14:04:01 Storage1001B kernel: [ 647.416165] block drbd1: asender
terminated
Dec 17 14:04:01 Storage1001B kernel: [ 647.416170] block drbd1: Terminating
drbd1_asender
Dec 17 14:04:01 Storage1001B kernel: [ 647.416223] block drbd1: Connection
closed
Dec 17 14:04:01 Storage1001B kernel: [ 647.416229] block drbd1: conn(
NetworkFailure -> Unconnected )
Dec 17 14:04:01 Storage1001B kernel: [ 647.416235] block drbd1: receiver
terminated
Dec 17 14:04:01 Storage1001B kernel: [ 647.416237] block drbd1: Restarting
drbd1_receiver
Dec 17 14:04:01 Storage1001B kernel: [ 647.416240] block drbd1: receiver
(re)started
Dec 17 14:04:01 Storage1001B kernel: [ 647.416244] block drbd1: conn(
Unconnected -> WFConnection )
Dec 17 14:04:04 Storage1001B cib: [3068]: notice: ais_dispatch_message:
Membership 1888: quorum lost
Dec 17 14:04:04 Storage1001B cib: [3068]: info: crm_update_peer: Node
Storage1001A.xdomain.com: id=191128586 state=lost (new) addr=r(0)
ip(10.100.100.11) votes=1 born=1864 seen=1884
proc=00000000000000000000000000111312
Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] notice:
pcmk_peer_update: Transitional membership event on ring 1888: memb=1, new=0,
lost=1
Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info:
pcmk_peer_update: memb: Storage1001B.xdomain.com 207905802
Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info:
pcmk_peer_update: lost: Storage1001A.xdomain.com 191128586
Dec 17 14:04:04 Storage1001B corosync[3033]: [VOTEQ ] quorum lost,
blocking activity
Dec 17 14:04:04 Storage1001B corosync[3033]: [QUORUM] This node is within
the non-primary component and will NOT provide any services.
Dec 17 14:04:04 Storage1001B corosync[3033]: [QUORUM] Members[1]:
207905802
Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] notice:
pcmk_peer_update: Stable membership event on ring 1888: memb=1, new=0,
lost=0
Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info:
pcmk_peer_update: MEMB: Storage1001B.xdomain.com 207905802
Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info:
ais_mark_unseen_peer_dead: Node Storage1001A.xdomain.com was not seen in the
previous transition
Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info: update_member:
Node 191128586/Storage1001A.xdomain.com is now: lost
Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: ais_dispatch_message:
Membership 1888: quorum lost
Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info:
send_member_notification: Sending membership update 1888 to 2 children
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: ais_status_callback:
status: Storage1001A.xdomain.com is now lost (was member)
Dec 17 14:04:04 Storage1001B corosync[3033]: [TOTEM ] A processor joined
or left the membership and a new membership was formed.
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: crm_update_peer: Node
Storage1001A.xdomain.com: id=191128586 state=lost (new) addr=r(0)
ip(10.100.100.11) votes=1 born=1864 seen=1884
proc=00000000000000000000000000111312
Dec 17 14:04:04 Storage1001B corosync[3033]: [CPG ] chosen downlist:
sender r(0) ip(10.100.100.12) ; members(old:2 left:1)
Dec 17 14:04:04 Storage1001B crmd: [3073]: WARN: check_dead_member: Our DC
node (Storage1001A.xdomain.com) left the cluster
Dec 17 14:04:04 Storage1001B corosync[3033]: [MAIN ] Completed service
synchronization, ready to provide service.
Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=check_dead_member ]
Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_te_control: Registering
TE UUID: aad28f35-6727-40aa-9b27-f0f51cc082a6
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: set_graph_functions:
Setting custom graph functions
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_dc_takeover: Taking over
DC status for this partition
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_readwrite: We
are now in R/W mode
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_master for section 'all' (origin=local/crmd/76,
version=0.261.9): ok (rc=0)
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/77,
version=0.261.10): ok (rc=0)
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/79, version=0.261.11): ok (rc=0)
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: join_make_offer: Making
join offers based on membership 1888
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/81, version=0.261.12): ok (rc=0)
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_dc_join_offer_all:
join-1: Waiting on 1 outstanding join acks
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: ais_dispatch_message:
Membership 1888: quorum still lost
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: crmd_ais_dispatch: Setting
expected votes to 2
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: ais_dispatch_message:
Membership 1888: quorum still lost
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/84, version=0.261.13): ok (rc=0)
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: crmd_ais_dispatch: Setting
expected votes to 2
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: update_dc: Set DC to
Storage1001B.xdomain.com (3.0.6)
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/86, version=0.261.14): ok (rc=0)
Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_dc_join_finalize:
join-1: Syncing the CIB from Storage1001B.xdomain.com to the rest of the
cluster
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_sync for section 'all' (origin=local/crmd/88,
version=0.261.14): ok (rc=0)
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/89,
version=0.261.15): ok (rc=0)
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_dc_join_ack: join-1:
Updating node state to member for Storage1001B.xdomain.com
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='Storage1001B.xdomain.com']/lrm
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001B.xdomain.com']/lrm (origin=local/crmd/90,
version=0.261.16): ok (rc=0)
Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p_drbd2:1 (10000)
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/92,
version=0.261.18): ok (rc=0)
Dec 17 14:04:04 Storage1001B crmd: [3073]: WARN: match_down_event: No match
for shutdown action on Storage1001A.xdomain.com
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: te_update_diff:
Stonith/shutdown of Storage1001A.xdomain.com not matched
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:234 - Triggered transition abort (complete=1, tag=node_state,
id=Storage1001A.xdomain.com, magic=NA, cib=0.261.19) : Node failure
Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/94,
version=0.261.20): ok (rc=0)
Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p_drbd1:1 (10000)
Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: p_ping (200)
Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node
Storage1001A.xdomain.com will be fenced because it is un-expectedly down
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: determine_online_status:
Node Storage1001A.xdomain.com is unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: unpack_rsc_op:
Processing failed op p_target_data01_last_failure_0 on
Storage1001A.xdomain.com: unknown exec error (-2)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: common_apply_stickiness:
Forcing p_target_data01 away from Storage1001A.xdomain.com after 1000000
failures (max=1000000)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_lvm01_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_ip1_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_ip11_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_target_data01_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_drbd1:0_demote_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_drbd1:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_drbd1:0_demote_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_drbd1:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_drbd2:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_drbd2:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_target:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action
p_ping:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking
node Storage1001A.xdomain.com unclean
Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node
Storage1001A.xdomain.com for STONITH
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Move
p_lvm01#011(Started Storage1001A.xdomain.com -> Storage1001B.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip1#011(Started Storage1001A.xdomain.com -> Storage1001B.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip11#011(Started Storage1001A.xdomain.com -> Storage1001B.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Move
p_target_data01#011(Started Storage1001A.xdomain.com ->
Storage1001B.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Start
p_lu_data01_vol01#011(Storage1001B.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Start
p_email_admin1#011(Storage1001B.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Demote
p_drbd1:0#011(Master -> Stopped Storage1001A.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd1:1#011(Slave -> Master Storage1001B.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_drbd2:0#011(Storage1001A.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_target:0#011(Storage1001A.xdomain.com)
Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_ping:0#011(Storage1001A.xdomain.com)
Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_te_invoke: Processing
graph 0 (ref=pe_calc-dc-1387317844-35) derived from
/var/lib/pengine/pe-warn-179.bz2
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 134: notify p_drbd1:1_pre_notify_demote_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:04:04 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[88]
(pid 14458)
Dec 17 14:04:04 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 138: notify p_drbd2:1_pre_notify_stop_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:04:04 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[89]
(pid 14459)
Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: te_fence_node: Executing
reboot fencing operation (108) on Storage1001A.xdomain.com (timeout=60000)
Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info:
initiate_remote_stonith_op: Initiating remote operation reboot for
Storage1001A.xdomain.com: 70f0c47c-e9be-470d-be4e-fb1da3f0587e
Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info:
can_fence_host_with_device: st_node_a can fence Storage1001A.xdomain.com:
static-list
Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info: call_remote_stonith:
Requesting that Storage1001B.xdomain.com perform op reboot
Storage1001A.xdomain.com
Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info:
can_fence_host_with_device: st_node_a can fence Storage1001A.xdomain.com:
static-list
Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info: stonith_fence: Found
1 matching devices for 'Storage1001A.xdomain.com'
Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info: stonith_command:
Processed st_fence from Storage1001B.xdomain.com: rc=-1
Dec 17 14:04:05 Storage1001B pengine: [3072]: WARN: process_pe_message:
Transition 0: WARNINGs found during PE processing. PEngine Input stored in:
/var/lib/pengine/pe-warn-179.bz2
Dec 17 14:04:05 Storage1001B pengine: [3072]: notice: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 17 14:04:05 Storage1001B lrmd: [3070]: info: operation notify[88] on
p_drbd1:1 for client 3073: pid 14458 exited with return code 0
Dec 17 14:04:05 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_notify_0 (call=88, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:04:05 Storage1001B lrmd: [3070]: info: operation notify[89] on
p_drbd2:1 for client 3073: pid 14459 exited with return code 0
Dec 17 14:04:05 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd2:1_notify_0 (call=89, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:04:05 Storage1001B stonith-ng: [3069]: notice: log_operation:
Operation 'reboot' [14133] (call 0 from
dd5b1efb-dcb7-4fcf-b41c-df8f0f1f97b0) for host 'Storage1001A.xdomain.com'
with device 'st_node_a' returned: 0
Dec 17 14:04:05 Storage1001B stonith-ng: [3069]: info: log_operation:
st_node_a: Performing: stonith -t external/synaccess -T reset Storage1001A
Dec 17 14:04:05 Storage1001B stonith-ng: [3069]: info: log_operation:
st_node_a: success: Storage1001A 0
Dec 17 14:04:05 Storage1001B stonith-ng: [3069]: notice: remote_op_done:
Operation reboot of Storage1001A.xdomain.com by Storage1001B.xdomain.com for
Storage1001A.xdomain.com[dd5b1efb-dcb7-4fcf-b41c-df8f0f1f97b0]: OK
Dec 17 14:04:05 Storage1001B crmd: [3073]: notice: tengine_stonith_notify:
Peer Storage1001A.xdomain.com was terminated (reboot) by
Storage1001B.xdomain.com for Storage1001A.xdomain.com: OK
(ref=be5efd32-362d-4c70-bdb8-9e27606cc748)
Dec 17 14:04:06 Storage1001B kernel: [ 652.076389] e1000e: eth5 NIC Link is
Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:04:06 Storage1001B kernel: [ 652.076396] e1000e 0000:07:00.1:
eth5: 10/100 speed: disabling TSO
Dec 17 14:04:06 Storage1001B kernel: [ 652.136019] bonding: bond1: link
status definitely up for interface eth5, 10 Mbps full duplex.
Dec 17 14:04:06 Storage1001B kernel: [ 652.136030] bonding: bond1: first
active interface up!
Dec 17 14:04:09 Storage1001B kernel: [ 655.420236] e1000e: eth5 NIC Link is
Down
Dec 17 14:04:09 Storage1001B kernel: [ 655.436019] bonding: bond1: link
status definitely down for interface eth5, disabling it
Dec 17 14:04:09 Storage1001B kernel: [ 655.436029] bonding: bond1: now
running without any active interface !
Dec 17 14:04:14 Storage1001B stonith-ng: [3069]: notice: log_operation:
Operation 'reboot' [14532] (call 0 from
06b688b5-7fbe-4e14-8f63-6705013bd929) for host 'Storage1001A.xdomain.com'
with device 'st_node_a' returned: 0
Dec 17 14:04:14 Storage1001B stonith-ng: [3069]: info: log_operation:
st_node_a: Performing: stonith -t external/synaccess -T reset Storage1001A
Dec 17 14:04:14 Storage1001B stonith-ng: [3069]: info: log_operation:
st_node_a: success: Storage1001A 0
Dec 17 14:04:14 Storage1001B stonith-ng: [3069]: notice: remote_op_done:
Operation reboot of Storage1001A.xdomain.com by Storage1001B.xdomain.com for
Storage1001B.xdomain.com[06b688b5-7fbe-4e14-8f63-6705013bd929]: OK
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: tengine_stonith_callback:
StonithOp <st-reply st_origin="stonith_construct_async_reply" t="stonith-ng"
st_op="reboot" st_remote_op="70f0c47c-e9be-470d-be4e-fb1da3f0587e"
st_clientid="06b688b5-7fbe-4e14-8f63-6705013bd929"
st_target="Storage1001A.xdomain.com" st_device_action="st_fence"
st_callid="0" st_callopt="0" st_rc="0" st_output="Performing: stonith -t
external/synaccess -T reset Storage1001A#012success:
Storage1001A 0#012" src="Storage1001B.xdomain.com" seq="6" state="2" />
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='Storage1001A.xdomain.com']/lrm
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='Storage1001A.xdomain.com']/transient_attributes
Dec 17 14:04:14 Storage1001B crmd: [3073]: notice: crmd_peer_update: Status
update: Client Storage1001A.xdomain.com/crmd now has status [offline]
(DC=true)
Dec 17 14:04:14 Storage1001B crmd: [3073]: notice: tengine_stonith_notify:
Peer Storage1001A.xdomain.com was terminated (reboot) by
Storage1001B.xdomain.com for Storage1001B.xdomain.com: OK
(ref=70f0c47c-e9be-470d-be4e-fb1da3f0587e)
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 114: notify p_drbd2:1_post_notify_stop_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[90]
(pid 14805)
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: cib_fencing_updated:
Fencing update 97 for Storage1001A.xdomain.com: complete
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 135: notify p_drbd1:1_post_notify_demote_0 on
Storage1001B.xdomain.com (local)
Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[91]
(pid 14806)
Dec 17 14:04:14 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001A.xdomain.com']/lrm (origin=local/crmd/98,
version=0.261.26): ok (rc=0)
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:320 - Triggered transition abort (complete=0, tag=lrm_rsc_op,
id=st_node_b_last_0, magic=0:7;5:0:7:6c3b329e-4284-439d-9d22-171f31f6deb7,
cib=0.261.26) : Resource op removal
Dec 17 14:04:14 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001A.xdomain.com']/transient_attributes
(origin=local/crmd/99, version=0.261.27): ok (rc=0)
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:194 - Triggered transition abort (complete=0,
tag=transient_attributes, id=Storage1001A.xdomain.com, magic=NA,
cib=0.261.27) : Transient attribute: removal
Dec 17 14:04:14 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p_drbd1:1 (1000)
Dec 17 14:04:14 Storage1001B attrd: [3071]: notice: attrd_perform_update:
Sent update 25: master-p_drbd1:1=1000
Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: RA output:
(p_drbd1:1:notify:stdout)
Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: operation notify[91] on
p_drbd1:1 for client 3073: pid 14806 exited with return code 0
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair,
id=status-Storage1001B.xdomain.com-master-p_drbd1.1, name=master-p_drbd1:1,
value=1000, magic=NA, cib=0.261.29) : Transient attribute: update
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_notify_0 (call=91, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: RA output:
(p_drbd2:1:notify:stdout)
Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: operation notify[90] on
p_drbd2:1 for client 3073: pid 14805 exited with return code 0
Dec 17 14:04:14 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd2:1_notify_0 (call=90, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:04:14 Storage1001B crmd: [3073]: notice: run_graph: ====
Transition 0 (Complete=36, Pending=0, Fired=0, Skipped=25, Incomplete=9,
Source=/var/lib/pengine/pe-warn-179.bz2): Stopped
Dec 17 14:04:14 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start
p_lvm01#011(Storage1001B.xdomain.com)
Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start
p_ip1#011(Storage1001B.xdomain.com)
Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start
p_ip11#011(Storage1001B.xdomain.com)
Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start
p_target_data01#011(Storage1001B.xdomain.com)
Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start
p_lu_data01_vol01#011(Storage1001B.xdomain.com)
Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start
p_email_admin1#011(Storage1001B.xdomain.com)
Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd1:1#011(Slave -> Master Storage1001B.xdomain.com)
Dec 17 14:04:14 Storage1001B kernel: [ 661.023431] e1000e: eth5 NIC Link is
Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:04:14 Storage1001B kernel: [ 661.023437] e1000e 0000:07:00.1:
eth5: 10/100 speed: disabling TSO
Dec 17 14:04:15 Storage1001B kernel: [ 661.036013] bonding: bond1: link
status definitely up for interface eth5, 10 Mbps full duplex.
Dec 17 14:04:15 Storage1001B kernel: [ 661.036023] bonding: bond1: first
active interface up!
Dec 17 14:04:15 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 17 14:04:15 Storage1001B crmd: [3073]: info: do_te_invoke: Processing
graph 1 (ref=pe_calc-dc-1387317854-44) derived from
/var/lib/pengine/pe-input-602.bz2
Dec 17 14:04:15 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 117: notify p_drbd1:1_pre_notify_promote_0 on
Storage1001B.xdomain.com (local)
Dec 17 14:04:15 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[92]
(pid 14861)
Dec 17 14:04:15 Storage1001B pengine: [3072]: notice: process_pe_message:
Transition 1: PEngine Input stored in: /var/lib/pengine/pe-input-602.bz2
Dec 17 14:04:15 Storage1001B lrmd: [3070]: info: operation notify[92] on
p_drbd1:1 for client 3073: pid 14861 exited with return code 0
Dec 17 14:04:15 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_notify_0 (call=92, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:04:15 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 31: promote p_drbd1:1_promote_0 on Storage1001B.xdomain.com (local)
Dec 17 14:04:15 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 promote[93]
(pid 14884)
Dec 17 14:04:15 Storage1001B kernel: [ 661.950685] block drbd1: helper
command: /sbin/drbdadm fence-peer minor-1
Dec 17 14:04:16 Storage1001B crm-fence-peer.sh[14931]: invoked for Data01
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: - <cib
admin_epoch="0" epoch="261" num_updates="29" />
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="262"
num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2"
crm_feature_set="3.0.6" update-origin="Storage1001A.xdomain.com"
update-client="crm_resource" cib-last-written="Tue Dec 17 14:03:36 2013"
have-quorum="0" dc-uuid="Storage1001B.xdomain.com" >
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <configuration
>
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <constraints
>
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: +
<rsc_location rsc="ms_drbd1" id="drbd-fence-by-handler-Data01-ms_drbd1"
__crm_diff_marker__="added:top" >
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <rule
role="Master" score="-INFINITY"
id="drbd-fence-by-handler-Data01-rule-ms_drbd1" >
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: +
<expression attribute="#uname" operation="ne"
value="Storage1001B.xdomain.com"
id="drbd-fence-by-handler-Data01-expr-ms_drbd1" />
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + </rule>
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: +
</rsc_location>
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: +
</constraints>
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: +
</configuration>
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + </cib>
Dec 17 14:04:16 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:126 - Triggered transition abort (complete=0, tag=diff,
id=(null), magic=NA, cib=0.262.1) : Non-status change
Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_create for section constraints
(origin=local/cibadmin/2, version=0.262.1): ok (rc=0)
Dec 17 14:04:16 Storage1001B crm-fence-peer.sh[14931]: INFO peer is not
reachable, my disk is UpToDate: placed constraint
'drbd-fence-by-handler-Data01-ms_drbd1'
Dec 17 14:04:16 Storage1001B kernel: [ 663.001061] block drbd1: helper
command: /sbin/drbdadm fence-peer minor-1 exit code 5 (0x500)
Dec 17 14:04:16 Storage1001B kernel: [ 663.001066] block drbd1: fence-peer
helper returned 5 (peer is unreachable, assumed to be dead)
Dec 17 14:04:16 Storage1001B kernel: [ 663.001077] block drbd1: role(
Secondary -> Primary ) pdsk( DUnknown -> Outdated )
Dec 17 14:04:16 Storage1001B kernel: [ 663.023104] block drbd1: new current
UUID E48A0D04194BDA73:82E61159C15A62E0:711FF2F50EF3A976:711EF2F50EF3A977
Dec 17 14:04:17 Storage1001B lrmd: [3070]: info: RA output:
(p_drbd1:1:promote:stdout)
Dec 17 14:04:17 Storage1001B lrmd: [3070]: info: operation promote[93] on
p_drbd1:1 for client 3073: pid 14884 exited with return code 0
Dec 17 14:04:17 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_promote_0 (call=93, rc=0, cib-update=102,
confirmed=true) ok
Dec 17 14:04:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 118: notify p_drbd1:1_post_notify_promote_0 on
Storage1001B.xdomain.com (local)
Dec 17 14:04:17 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[94]
(pid 14960)
Dec 17 14:04:18 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p_drbd1:1 (10000)
Dec 17 14:04:18 Storage1001B attrd: [3071]: notice: attrd_perform_update:
Sent update 27: master-p_drbd1:1=10000
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation notify[94] on
p_drbd1:1 for client 3073: pid 14960 exited with return code 0
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: RA output:
(p_drbd1:1:notify:stdout)
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair,
id=status-Storage1001B.xdomain.com-master-p_drbd1.1, name=master-p_drbd1:1,
value=10000, magic=NA, cib=0.262.3) : Transient attribute: update
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_notify_0 (call=94, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:04:18 Storage1001B crmd: [3073]: notice: run_graph: ====
Transition 1 (Complete=9, Pending=0, Fired=0, Skipped=14, Incomplete=0,
Source=/var/lib/pengine/pe-input-602.bz2): Stopped
Dec 17 14:04:18 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start
p_lvm01#011(Storage1001B.xdomain.com)
Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start
p_ip1#011(Storage1001B.xdomain.com)
Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start
p_ip11#011(Storage1001B.xdomain.com)
Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start
p_target_data01#011(Storage1001B.xdomain.com)
Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start
p_lu_data01_vol01#011(Storage1001B.xdomain.com)
Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start
p_email_admin1#011(Storage1001B.xdomain.com)
Dec 17 14:04:18 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: do_te_invoke: Processing
graph 2 (ref=pe_calc-dc-1387317858-50) derived from
/var/lib/pengine/pe-input-603.bz2
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 9: start p_lvm01_start_0 on Storage1001B.xdomain.com (local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_lvm01 start[95] (pid
15023)
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 32: monitor p_drbd1:1_monitor_3000 on Storage1001B.xdomain.com
(local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 monitor[96]
(pid 15024)
Dec 17 14:04:18 Storage1001B LVM[15023]: INFO: Activating volume group
vg_data01
Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: process_pe_message:
Transition 2: PEngine Input stored in: /var/lib/pengine/pe-input-603.bz2
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[96] on
p_drbd1:1 for client 3073: pid 15024 exited with return code 8
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_monitor_3000 (call=96, rc=8, cib-update=104,
confirmed=false) master
Dec 17 14:04:18 Storage1001B LVM[15023]: INFO: Reading all physical volumes.
This may take a while... Found volume group "Storage1001B" using metadata
type lvm2 Found volume group "vg_data02" using metadata type lvm2 Found
volume group "vg_data01" using metadata type lvm2
Dec 17 14:04:18 Storage1001B LVM[15023]: INFO: 1 logical volume(s) in volume
group "vg_data01" now active
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[95] on
p_lvm01 for client 3073: pid 15023 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_lvm01_start_0 (call=95, rc=0, cib-update=105, confirmed=true) ok
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 10: monitor p_lvm01_monitor_4000 on Storage1001B.xdomain.com (local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_lvm01 monitor[97]
(pid 15091)
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 11: start p_ip1_start_0 on Storage1001B.xdomain.com (local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_ip1 start[98] (pid
15092)
Dec 17 14:04:18 Storage1001B IPaddr[15092]: INFO: Using calculated netmask
for 10.11.2.13: 255.255.248.0
Dec 17 14:04:18 Storage1001B IPaddr[15092]: INFO: eval ifconfig bond0:0
10.11.2.13 netmask 255.255.248.0 broadcast 10.11.7.255
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[98] on
p_ip1 for client 3073: pid 15092 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_ip1_start_0 (call=98, rc=0, cib-update=106, confirmed=true) ok
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 12: monitor p_ip1_monitor_5000 on Storage1001B.xdomain.com (local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_ip1 monitor[99] (pid
15182)
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 13: start p_ip11_start_0 on Storage1001B.xdomain.com (local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_ip11 start[100] (pid
15183)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[99] on
p_ip1 for client 3073: pid 15182 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_ip1_monitor_5000 (call=99, rc=0, cib-update=107,
confirmed=false) ok
Dec 17 14:04:18 Storage1001B IPaddr[15183]: INFO: Using calculated netmask
for 10.11.10.13: 255.255.248.0
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[97] on
p_lvm01 for client 3073: pid 15091 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_lvm01_monitor_4000 (call=97, rc=0, cib-update=108,
confirmed=false) ok
Dec 17 14:04:18 Storage1001B IPaddr[15183]: INFO: eval ifconfig bond2:0
10.11.10.13 netmask 255.255.248.0 broadcast 10.11.15.255
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[100] on
p_ip11 for client 3073: pid 15183 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_ip11_start_0 (call=100, rc=0, cib-update=109, confirmed=true) ok
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 14: monitor p_ip11_monitor_5000 on Storage1001B.xdomain.com (local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_ip11 monitor[101]
(pid 15268)
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 15: start p_target_data01_start_0 on Storage1001B.xdomain.com (local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_target_data01
start[102] (pid 15269)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[101] on
p_ip11 for client 3073: pid 15268 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_ip11_monitor_5000 (call=101, rc=0, cib-update=110,
confirmed=false) ok
Dec 17 14:04:18 Storage1001B iSCSITarget[15269]: INFO: Successfully created
network portal: 0.0.0.0:3260 created
iqn.2013-10.com.xdomain.storage1001.data01 TPGT: 1
Dec 17 14:04:18 Storage1001B iSCSITarget[15269]: INFO: Successfully disabled
Initiator ACL mode (Enabled DemoMode) on iSCSI Target Portal Group:
iqn.2013-10.com.xdomain.storage1001.data01 1
Dec 17 14:04:18 Storage1001B iSCSITarget[15269]: INFO: Successfully disabled
iSCSI Authentication on iSCSI Target Portal Group:
iqn.2013-10.com.xdomain.storage1001.data01 1
Dec 17 14:04:18 Storage1001B iSCSITarget[15269]: INFO: Successfully enabled
iSCSI Target Portal Group: iqn.2013-10.com.xdomain.storage1001.data01 1
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[102] on
p_target_data01 for client 3073: pid 15269 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_target_data01_start_0 (call=102, rc=0, cib-update=111,
confirmed=true) ok
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 16: monitor p_target_data01_monitor_10000 on Storage1001B.xdomain.com
(local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_target_data01
monitor[103] (pid 15367)
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 17: start p_lu_data01_vol01_start_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_lu_data01_vol01
start[104] (pid 15368)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[103] on
p_target_data01 for client 3073: pid 15367 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_target_data01_monitor_10000 (call=103, rc=0, cib-update=112,
confirmed=false) ok
Dec 17 14:04:18 Storage1001B kernel: [ 664.698063] bio: create slab <bio-1>
at 1
Dec 17 14:04:18 Storage1001B iSCSILogicalUnit[15368]: INFO: Status:
DEACTIVATED Execute/Left/Max Queue Depth: 0/128/128 SectorSize: 512
MaxSectors: 256 iBlock device: dm-2 UDEV PATH: /dev/vg_data01/lv_vol01
Major: 254 Minor: 2 CLAIMED: IBLOCK
Dec 17 14:04:18 Storage1001B iSCSILogicalUnit[15368]: INFO: Successfully
created iSCSI Target Logical Unit
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[104] on
p_lu_data01_vol01 for client 3073: pid 15368 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_lu_data01_vol01_start_0 (call=104, rc=0, cib-update=113,
confirmed=true) ok
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 18: monitor p_lu_data01_vol01_monitor_10000 on
Storage1001B.xdomain.com (local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_lu_data01_vol01
monitor[105] (pid 15486)
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 19: start p_email_admin1_start_0 on Storage1001B.xdomain.com (local)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_email_admin1
start[106] (pid 15487)
Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[105] on
p_lu_data01_vol01 for client 3073: pid 15486 exited with return code 0
Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_lu_data01_vol01_monitor_10000 (call=105, rc=0, cib-update=114,
confirmed=false) ok
Dec 17 14:04:19 Storage1001B lrmd: [3070]: info: operation start[106] on
p_email_admin1 for client 3073: pid 15487 exited with return code 0
Dec 17 14:04:19 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_email_admin1_start_0 (call=106, rc=0, cib-update=115,
confirmed=true) ok
Dec 17 14:04:19 Storage1001B crmd: [3073]: notice: run_graph: ====
Transition 2 (Complete=14, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-603.bz2): Complete
Dec 17 14:04:19 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 17 14:04:19 Storage1001B kernel: [ 665.824233] e1000e: eth5 NIC Link is
Down
Dec 17 14:04:19 Storage1001B kernel: [ 665.836015] bonding: bond1: link
status definitely down for interface eth5, disabling it
Dec 17 14:04:19 Storage1001B kernel: [ 665.836024] bonding: bond1: now
running without any active interface !
Dec 17 14:04:21 Storage1001B kernel: [ 667.416923] e1000e: eth1 NIC Link is
Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:04:21 Storage1001B kernel: [ 667.416930] e1000e 0000:05:00.1:
eth1: 10/100 speed: disabling TSO
Dec 17 14:04:21 Storage1001B kernel: [ 667.436017] bonding: bond1: link
status definitely up for interface eth1, 10 Mbps full duplex.
Dec 17 14:04:21 Storage1001B kernel: [ 667.436030] bonding: bond1: first
active interface up!
Dec 17 14:04:21 Storage1001B kernel: [ 667.671427] e1000e: eth5 NIC Link is
Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:04:21 Storage1001B kernel: [ 667.671433] e1000e 0000:07:00.1:
eth5: 10/100 speed: disabling TSO
Dec 17 14:04:21 Storage1001B kernel: [ 667.736011] bonding: bond1: link
status definitely up for interface eth5, 10 Mbps full duplex.
Dec 17 14:04:22 Storage1001B ntpd[2540]: Listen normally on 14 bond0:0
10.11.2.13 UDP 123
Dec 17 14:04:22 Storage1001B ntpd[2540]: Listen normally on 15 bond2:0
10.11.10.13 UDP 123
Dec 17 14:04:22 Storage1001B ntpd[2540]: peers refreshed
Dec 17 14:04:31 Storage1001B kernel: [ 677.592125] e1000e: eth1 NIC Link is
Down
Dec 17 14:04:31 Storage1001B kernel: [ 677.636014] bonding: bond1: link
status definitely down for interface eth1, disabling it
Dec 17 14:04:31 Storage1001B kernel: [ 677.780230] e1000e: eth5 NIC Link is
Down
Dec 17 14:04:31 Storage1001B kernel: [ 677.836011] bonding: bond1: link
status definitely down for interface eth5, disabling it
Dec 17 14:04:31 Storage1001B kernel: [ 677.836018] bonding: bond1: now
running without any active interface !
Dec 17 14:04:33 Storage1001B kernel: [ 679.808970] e1000e: eth1 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:04:33 Storage1001B kernel: [ 679.836012] bonding: bond1: link
status definitely up for interface eth1, 1000 Mbps full duplex.
Dec 17 14:04:33 Storage1001B kernel: [ 679.836023] bonding: bond1: first
active interface up!
Dec 17 14:04:34 Storage1001B kernel: [ 680.543355] e1000e: eth5 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:04:34 Storage1001B kernel: [ 680.636010] bonding: bond1: link
status definitely up for interface eth5, 1000 Mbps full duplex.
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="263"
num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2"
crm_feature_set="3.0.6" update-origin="Storage1001B.xdomain.com"
update-client="cibadmin" cib-last-written="Tue Dec 17 14:04:16 2013"
have-quorum="1" dc-uuid="Storage1001B.xdomain.com" />
Dec 17 14:05:01 Storage1001B /USR/SBIN/CRON[17000]: (root) CMD (command -v
debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 17 14:05:16 Storage1001B kernel: [ 722.400128] e1000e: eth1 NIC Link is
Down
Dec 17 14:05:16 Storage1001B kernel: [ 722.436017] bonding: bond1: link
status definitely down for interface eth1, disabling it
Dec 17 14:05:16 Storage1001B kernel: [ 722.620238] e1000e: eth5 NIC Link is
Down
Dec 17 14:05:16 Storage1001B kernel: [ 722.636008] bonding: bond1: link
status definitely down for interface eth5, disabling it
Dec 17 14:05:16 Storage1001B kernel: [ 722.636013] bonding: bond1: now
running without any active interface !
Dec 17 14:05:18 Storage1001B kernel: [ 725.023327] e1000e: eth5 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:05:19 Storage1001B kernel: [ 725.036023] bonding: bond1: link
status definitely up for interface eth5, 1000 Mbps full duplex.
Dec 17 14:05:19 Storage1001B kernel: [ 725.036036] bonding: bond1: first
active interface up!
Dec 17 14:05:29 Storage1001B kernel: [ 735.816294] e1000e: eth5 NIC Link is
Down
Dec 17 14:05:29 Storage1001B kernel: [ 735.836012] bonding: bond1: link
status definitely down for interface eth5, disabling it
Dec 17 14:05:29 Storage1001B kernel: [ 735.836021] bonding: bond1: now
running without any active interface !
Dec 17 14:05:31 Storage1001B kernel: [ 737.220924] e1000e: eth1 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:05:31 Storage1001B kernel: [ 737.236010] bonding: bond1: link
status definitely up for interface eth1, 1000 Mbps full duplex.
Dec 17 14:05:31 Storage1001B kernel: [ 737.236021] bonding: bond1: first
active interface up!
Dec 17 14:05:32 Storage1001B kernel: [ 738.343456] e1000e: eth5 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 17 14:05:32 Storage1001B kernel: [ 738.436013] bonding: bond1: link
status definitely up for interface eth5, 1000 Mbps full duplex.
Dec 17 14:05:43 Storage1001B kernel: [ 749.272055] block drbd1: Handshake
successful: Agreed network protocol version 96
Dec 17 14:05:43 Storage1001B kernel: [ 749.272068] block drbd2: Handshake
successful: Agreed network protocol version 96
Dec 17 14:05:43 Storage1001B kernel: [ 749.272074] block drbd1: conn(
WFConnection -> WFReportParams )
Dec 17 14:05:43 Storage1001B kernel: [ 749.272088] block drbd2: conn(
WFConnection -> WFReportParams )
Dec 17 14:05:43 Storage1001B kernel: [ 749.272120] block drbd2: Starting
asender thread (from drbd2_receiver [2983])
Dec 17 14:05:43 Storage1001B kernel: [ 749.272165] block drbd1: Starting
asender thread (from drbd1_receiver [2980])
Dec 17 14:05:43 Storage1001B kernel: [ 749.272283] block drbd1:
data-integrity-alg: <not-used>
Dec 17 14:05:43 Storage1001B kernel: [ 749.272303] block drbd1:
drbd_sync_handshake:
Dec 17 14:05:43 Storage1001B kernel: [ 749.272306] block drbd1: self
E48A0D04194BDA73:82E61159C15A62E0:711FF2F50EF3A976:711EF2F50EF3A977 bits:0
flags:0
Dec 17 14:05:43 Storage1001B kernel: [ 749.272309] block drbd1: peer
82E61159C15A62E0:0000000000000000:711FF2F50EF3A977:711EF2F50EF3A977
bits:37888 flags:2
Dec 17 14:05:43 Storage1001B kernel: [ 749.272313] block drbd1:
uuid_compare()=1 by rule 70
Dec 17 14:05:43 Storage1001B kernel: [ 749.272319] block drbd1: peer(
Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated ->
Consistent )
Dec 17 14:05:43 Storage1001B kernel: [ 749.272560] block drbd2:
data-integrity-alg: <not-used>
Dec 17 14:05:43 Storage1001B kernel: [ 749.272582] block drbd2:
drbd_sync_handshake:
Dec 17 14:05:43 Storage1001B kernel: [ 749.272585] block drbd2: self
2A0715EFC37BB5BB:76A87E2130121407:B007DA0B699331F9:B006DA0B699331F9 bits:0
flags:0
Dec 17 14:05:43 Storage1001B kernel: [ 749.272588] block drbd2: peer
76A87E2130121406:0000000000000000:B007DA0B699331F8:B006DA0B699331F9 bits:0
flags:0
Dec 17 14:05:43 Storage1001B kernel: [ 749.272592] block drbd2:
uuid_compare()=1 by rule 70
Dec 17 14:05:43 Storage1001B kernel: [ 749.272603] block drbd2: peer(
Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated ->
Consistent )
Dec 17 14:05:44 Storage1001B kernel: [ 750.082878] block drbd1: helper
command: /sbin/drbdadm before-resync-source minor-1
Dec 17 14:05:44 Storage1001B kernel: [ 750.086021] block drbd1: helper
command: /sbin/drbdadm before-resync-source minor-1 exit code 0 (0x0)
Dec 17 14:05:44 Storage1001B kernel: [ 750.086033] block drbd1: conn(
WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
Dec 17 14:05:44 Storage1001B kernel: [ 750.086043] block drbd1: Began
resync as SyncSource (will sync 151552 KB [37888 bits set]).
Dec 17 14:05:44 Storage1001B kernel: [ 750.086065] block drbd1: updated
sync UUID
E48A0D04194BDA73:82E71159C15A62E0:82E61159C15A62E0:711FF2F50EF3A976
Dec 17 14:05:44 Storage1001B cib: [3068]: notice: ais_dispatch_message:
Membership 1892: quorum acquired
Dec 17 14:05:44 Storage1001B cib: [3068]: info: crm_update_peer: Node
Storage1001A.xdomain.com: id=191128586 state=member (new) addr=r(0)
ip(10.100.100.11) votes=1 born=1864 seen=1892
proc=00000000000000000000000000111312
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] notice:
pcmk_peer_update: Transitional membership event on ring 1892: memb=1, new=0,
lost=0
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info:
pcmk_peer_update: memb: Storage1001B.xdomain.com 207905802
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] notice:
pcmk_peer_update: Stable membership event on ring 1892: memb=2, new=1,
lost=0
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: update_member:
Node 191128586/Storage1001A.xdomain.com is now: member
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info:
pcmk_peer_update: NEW: Storage1001A.xdomain.com 191128586
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info:
pcmk_peer_update: MEMB: Storage1001A.xdomain.com 191128586
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info:
pcmk_peer_update: MEMB: Storage1001B.xdomain.com 207905802
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info:
send_member_notification: Sending membership update 1892 to 2 children
Dec 17 14:05:44 Storage1001B corosync[3033]: [TOTEM ] A processor joined
or left the membership and a new membership was formed.
Dec 17 14:05:44 Storage1001B crmd: [3073]: notice: ais_dispatch_message:
Membership 1892: quorum acquired
Dec 17 14:05:44 Storage1001B crmd: [3073]: notice: crmd_peer_update: Status
update: Client Storage1001A.xdomain.com/crmd now has status [online]
(DC=true)
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: ais_status_callback:
status: Storage1001A.xdomain.com is now member (was lost)
Dec 17 14:05:44 Storage1001B crmd: [3073]: WARN: match_down_event: No match
for shutdown action on Storage1001A.xdomain.com
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='Storage1001A.xdomain.com']/lrm
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='Storage1001A.xdomain.com']/transient_attributes
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: crm_update_peer: Node
Storage1001A.xdomain.com: id=191128586 state=member (new) addr=r(0)
ip(10.100.100.11) votes=1 born=1864 seen=1892
proc=00000000000000000000000000111312 (new)
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001A.xdomain.com']/lrm (origin=local/crmd/117,
version=0.262.17): ok (rc=0)
Dec 17 14:05:44 Storage1001B corosync[3033]: [VOTEQ ] quorum regained,
resuming activity
Dec 17 14:05:44 Storage1001B corosync[3033]: [QUORUM] This node is within
the primary component and will provide service.
Dec 17 14:05:44 Storage1001B corosync[3033]: [QUORUM] Members[2]:
191128586 207905802
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: update_member:
0x2561200 Node 191128586 (Storage1001A.xdomain.com) born on: 1892
Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info:
send_member_notification: Sending membership update 1892 to 2 children
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001A.xdomain.com']/transient_attributes
(origin=local/crmd/118, version=0.262.18): ok (rc=0)
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/119,
version=0.262.19): ok (rc=0)
Dec 17 14:05:44 Storage1001B cib: [3068]: info: ais_dispatch_message:
Membership 1892: quorum retained
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/121,
version=0.262.21): ok (rc=0)
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: crmd_ais_dispatch: Setting
expected votes to 2
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: ais_dispatch_message:
Membership 1892: quorum retained
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/123, version=0.262.22): ok (rc=0)
Dec 17 14:05:44 Storage1001B kernel: [ 750.300208] block drbd2: helper
command: /sbin/drbdadm before-resync-source minor-2
Dec 17 14:05:44 Storage1001B kernel: [ 750.303982] block drbd2: helper
command: /sbin/drbdadm before-resync-source minor-2 exit code 0 (0x0)
Dec 17 14:05:44 Storage1001B kernel: [ 750.303993] block drbd2: conn(
WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
Dec 17 14:05:44 Storage1001B kernel: [ 750.304018] block drbd2: Began
resync as SyncSource (will sync 0 KB [0 bits set]).
Dec 17 14:05:44 Storage1001B kernel: [ 750.304027] block drbd2: updated
sync UUID
2A0715EFC37BB5BB:76A97E2130121407:76A87E2130121407:B007DA0B699331F9
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/124,
version=0.262.23): ok (rc=0)
Dec 17 14:05:44 Storage1001B corosync[3033]: [CPG ] chosen downlist:
sender r(0) ip(10.100.100.11) ; members(old:1 left:0)
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: crmd_ais_dispatch: Setting
expected votes to 2
Dec 17 14:05:44 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_FSA_INTERNAL origin=crmd_peer_update ]
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: abort_transition_graph:
do_te_invoke:169 - Triggered transition abort (complete=1) : Peer Halt
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: join_make_offer: Making
join offers based on membership 1892
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: do_dc_join_offer_all:
join-2: Waiting on 2 outstanding join acks
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/127, version=0.262.25): ok (rc=0)
Dec 17 14:05:44 Storage1001B corosync[3033]: [MAIN ] Completed service
synchronization, ready to provide service.
Dec 17 14:05:44 Storage1001B crmd: [3073]: info: update_dc: Set DC to
Storage1001B.xdomain.com (3.0.6)
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <cib
admin_epoch="0" epoch="262" num_updates="25" >
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <configuration
>
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <constraints
>
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: -
<rsc_location rsc="ms_drbd2" id="drbd-fence-by-handler-Data02-ms_drbd2"
__crm_diff_marker__="removed:top" >
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <rule
role="Master" score="-INFINITY"
id="drbd-fence-by-handler-Data02-rule-ms_drbd2" >
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: -
<expression attribute="#uname" operation="ne"
value="Storage1001B.xdomain.com"
id="drbd-fence-by-handler-Data02-expr-ms_drbd2" />
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - </rule>
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: -
</rsc_location>
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: -
</constraints>
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: -
</configuration>
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - </cib>
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="263"
num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2"
crm_feature_set="3.0.6" update-origin="Storage1001B.xdomain.com"
update-client="cibadmin" cib-last-written="Tue Dec 17 14:04:16 2013"
have-quorum="1" dc-uuid="Storage1001B.xdomain.com" />
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section 'all'
(origin=Storage1001A.xdomain.com/cibadmin/2, version=0.263.1): ok (rc=0)
Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_sync_one for section 'all'
(origin=Storage1001A.xdomain.com/Storage1001A.xdomain.com/(null),
version=0.263.1): ok (rc=0)
Dec 17 14:05:44 Storage1001B kernel: [ 750.940922] block drbd2: Resync done
(total 1 sec; paused 0 sec; 0 K/sec)
Dec 17 14:05:44 Storage1001B kernel: [ 750.940929] block drbd2: updated
UUIDs 2A0715EFC37BB5BB:0000000000000000:76A97E2130121407:76A87E2130121407
Dec 17 14:05:44 Storage1001B kernel: [ 750.940937] block drbd2: conn(
SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
Dec 17 14:05:45 Storage1001B kernel: [ 751.403368] block drbd2: bitmap
WRITE of 14901 pages took 115 jiffies
Dec 17 14:05:45 Storage1001B crmd: [3073]: info: do_dc_join_offer_all: A new
node joined the cluster
Dec 17 14:05:45 Storage1001B crmd: [3073]: info: do_dc_join_offer_all:
join-3: Waiting on 2 outstanding join acks
Dec 17 14:05:45 Storage1001B crmd: [3073]: info: update_dc: Set DC to
Storage1001B.xdomain.com (3.0.6)
Dec 17 14:05:45 Storage1001B kernel: [ 751.572257] block drbd2: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Dec 17 14:05:46 Storage1001B kernel: [ 752.425119] block drbd1: Resync done
(total 2 sec; paused 0 sec; 75776 K/sec)
Dec 17 14:05:46 Storage1001B kernel: [ 752.425126] block drbd1: updated
UUIDs E48A0D04194BDA73:0000000000000000:82E71159C15A62E0:82E61159C15A62E0
Dec 17 14:05:46 Storage1001B kernel: [ 752.425135] block drbd1: conn(
SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <cib
admin_epoch="0" epoch="263" num_updates="1" >
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <configuration
>
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <constraints
>
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: -
<rsc_location rsc="ms_drbd1" id="drbd-fence-by-handler-Data01-ms_drbd1"
__crm_diff_marker__="removed:top" >
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <rule
role="Master" score="-INFINITY"
id="drbd-fence-by-handler-Data01-rule-ms_drbd1" >
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: -
<expression attribute="#uname" operation="ne"
value="Storage1001B.xdomain.com"
id="drbd-fence-by-handler-Data01-expr-ms_drbd1" />
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - </rule>
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: -
</rsc_location>
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: -
</constraints>
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: -
</configuration>
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - </cib>
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="264"
num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2"
crm_feature_set="3.0.6" update-origin="Storage1001A.xdomain.com"
update-client="cibadmin" cib-last-written="Tue Dec 17 14:05:44 2013"
have-quorum="1" dc-uuid="Storage1001B.xdomain.com" />
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section 'all'
(origin=Storage1001A.xdomain.com/cibadmin/2, version=0.264.1): ok (rc=0)
Dec 17 14:05:46 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Dec 17 14:05:46 Storage1001B crmd: [3073]: info: do_dc_join_finalize:
join-3: Syncing the CIB from Storage1001A.xdomain.com to the rest of the
cluster
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=Storage1001A.xdomain.com/Storage1001A.xdomain.com/130,
version=0.264.1): ok (rc=0)
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/131,
version=0.264.2): ok (rc=0)
Dec 17 14:05:46 Storage1001B kernel: [ 752.571688] block drbd1: bitmap
WRITE of 14889 pages took 23 jiffies
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/132,
version=0.264.3): ok (rc=0)
Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001A.xdomain.com']/transient_attributes
(origin=Storage1001A.xdomain.com/crmd/6, version=0.264.4): ok (rc=0)
Dec 17 14:05:46 Storage1001B kernel: [ 752.775379] block drbd1: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: do_dc_join_ack: join-3:
Updating node state to member for Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='Storage1001A.xdomain.com']/lrm
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: do_dc_join_ack: join-3:
Updating node state to member for Storage1001B.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='Storage1001B.xdomain.com']/lrm
Dec 17 14:05:47 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001A.xdomain.com']/lrm (origin=local/crmd/133,
version=0.264.5): ok (rc=0)
Dec 17 14:05:47 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p_drbd2:1 (10000)
Dec 17 14:05:47 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001B.xdomain.com']/lrm (origin=local/crmd/135,
version=0.264.7): ok (rc=0)
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op,
id=st_node_b_last_0, magic=0:7;9:399:7:6c3b329e-4284-439d-9d22-171f31f6deb7,
cib=0.264.7) : Resource op removal
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:276 - Triggered transition abort (complete=1, tag=diff,
id=(null), magic=NA, cib=0.264.8) : LRM Refresh
Dec 17 14:05:47 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/137,
version=0.264.9): ok (rc=0)
Dec 17 14:05:47 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/139,
version=0.264.11): ok (rc=0)
Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p_drbd1:1 (10000)
Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: p_ping (200)
Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: LogActions: Start
p_drbd1:0#011(Storage1001A.xdomain.com)
Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: LogActions: Start
p_drbd2:0#011(Storage1001A.xdomain.com)
Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: LogActions: Start
p_target:0#011(Storage1001A.xdomain.com)
Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: LogActions: Start
p_ping:0#011(Storage1001A.xdomain.com)
Dec 17 14:05:47 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: do_te_invoke: Processing
graph 3 (ref=pe_calc-dc-1387317947-74) derived from
/var/lib/pengine/pe-input-604.bz2
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 14: monitor st_node_a_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 15: monitor st_node_b_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 16: monitor p_lvm01_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 17: monitor p_ip1_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 18: monitor p_ip11_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 19: monitor p_target_data01_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 20: monitor p_lu_data01_vol01_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 21: monitor p_email_admin1_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 22: monitor p_lvm02_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 23: monitor p_ip2_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 24: monitor p_ip22_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 25: monitor p_target_data02_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 26: monitor p_lu_data02_vol01_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 27: monitor p_email_admin2_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 28: monitor p_drbd1:0_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 29: monitor p_drbd2:0_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 30: monitor p_target:0_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 31: monitor p_ping:0_monitor_0 on Storage1001A.xdomain.com
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 143: notify p_drbd1:1_pre_notify_start_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[107]
(pid 18654)
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 154: notify p_drbd2:1_pre_notify_start_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[108]
(pid 18655)
Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: process_pe_message:
Transition 3: PEngine Input stored in: /var/lib/pengine/pe-input-604.bz2
Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: RA output:
(p_drbd2:1:notify:stdout)
Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: RA output:
(p_drbd1:1:notify:stdout)
Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: operation notify[108] on
p_drbd2:1 for client 3073: pid 18655 exited with return code 0
Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: operation notify[107] on
p_drbd1:1 for client 3073: pid 18654 exited with return code 0
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd2:1_notify_0 (call=108, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_notify_0 (call=107, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:05:47 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 19
(p_target_data01_monitor_0) on Storage1001A.xdomain.com failed (target: 7
vs. rc: 0): Error
Dec 17 14:05:47 Storage1001B crmd: [3073]: info: abort_transition_graph:
match_graph_event:277 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=p_target_data01_last_failure_0,
magic=0:0;19:3:7:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.20) : Event
failed
Dec 17 14:05:49 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 28
(p_drbd1:0_monitor_0) on Storage1001A.xdomain.com failed (target: 7 vs. rc:
0): Error
Dec 17 14:05:49 Storage1001B crmd: [3073]: info: abort_transition_graph:
match_graph_event:277 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=p_drbd1:0_last_failure_0,
magic=0:0;28:3:7:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.29) : Event
failed
Dec 17 14:05:49 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair,
id=status-Storage1001A.xdomain.com-master-p_drbd1.0, name=master-p_drbd1:0,
value=10000, magic=NA, cib=0.264.30) : Transient attribute: update
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair,
id=status-Storage1001A.xdomain.com-master-p_drbd2.0, name=master-p_drbd2:0,
value=10000, magic=NA, cib=0.264.33) : Transient attribute: update
Dec 17 14:05:50 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 29
(p_drbd2:0_monitor_0) on Storage1001A.xdomain.com failed (target: 7 vs. rc:
0): Error
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: abort_transition_graph:
match_graph_event:277 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=p_drbd2:0_last_failure_0,
magic=0:0;29:3:7:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.34) : Event
failed
Dec 17 14:05:50 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 30
(p_target:0_monitor_0) on Storage1001A.xdomain.com failed (target: 7 vs. rc:
0): Error
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: abort_transition_graph:
match_graph_event:277 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=p_target:0_last_failure_0,
magic=0:0;30:3:7:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.35) : Event
failed
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 13: probe_complete probe_complete on Storage1001A.xdomain.com - no
waiting
Dec 17 14:05:50 Storage1001B crmd: [3073]: notice: run_graph: ====
Transition 3 (Complete=29, Pending=0, Fired=0, Skipped=10, Incomplete=12,
Source=/var/lib/pengine/pe-input-604.bz2): Stopped
Dec 17 14:05:50 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Dec 17 14:05:50 Storage1001B pengine: [3072]: ERROR: native_create_actions:
Resource p_target_data01 (ocf::iSCSITarget) is active on 2 nodes attempting
recovery
Dec 17 14:05:50 Storage1001B pengine: [3072]: WARN: native_create_actions:
See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more
information.
Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: LogActions: Move
p_target_data01#011(Started Storage1001A.xdomain.com ->
Storage1001B.xdomain.com)
Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: LogActions: Restart
p_lu_data01_vol01#011(Started Storage1001B.xdomain.com)
Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: LogActions: Restart
p_email_admin1#011(Started Storage1001B.xdomain.com)
Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: LogActions: Start
p_ping:0#011(Storage1001A.xdomain.com)
Dec 17 14:05:50 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: do_te_invoke: Processing
graph 4 (ref=pe_calc-dc-1387317950-98) derived from
/var/lib/pengine/pe-error-0.bz2
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 27: stop p_email_admin1_stop_0 on Storage1001B.xdomain.com (local)
Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: rsc:p_email_admin1
stop[109] (pid 18886)
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 40: monitor p_drbd1:0_monitor_4000 on Storage1001A.xdomain.com
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 72: monitor p_drbd2:0_monitor_4000 on Storage1001A.xdomain.com
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 103: monitor p_target:0_monitor_30000 on Storage1001A.xdomain.com
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 13: probe_complete probe_complete on Storage1001A.xdomain.com - no
waiting
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 110: start p_ping:0_start_0 on Storage1001A.xdomain.com
Dec 17 14:05:50 Storage1001B pengine: [3072]: ERROR: process_pe_message:
Transition 4: ERRORs found during PE processing. PEngine Input stored in:
/var/lib/pengine/pe-error-0.bz2
Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: operation stop[109] on
p_email_admin1 for client 3073: pid 18886 exited with return code 0
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_email_admin1_stop_0 (call=109, rc=0, cib-update=144,
confirmed=true) ok
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 25: stop p_lu_data01_vol01_stop_0 on Storage1001B.xdomain.com (local)
Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: cancel_op: operation
monitor[105] on p_lu_data01_vol01 for client 3073, its parameters:
implementation=[lio] CRM_meta_timeout=[20000] CRM_meta_name=[monitor]
target_iqn=[iqn.2013-10.com.xdomain.storage1001.data01]
path=[/dev/vg_data01/lv_vol01] crm_feature_set=[3.0.6]
CRM_meta_interval=[10000] lun=[1] cancelled
Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: rsc:p_lu_data01_vol01
stop[110] (pid 18902)
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_lu_data01_vol01_monitor_10000 (call=105, status=1, cib-update=0,
confirmed=true) Cancelled
Dec 17 14:05:50 Storage1001B iSCSILogicalUnit[18902]: INFO: Successfully
deleted iSCSI Target Logical Unit
Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: operation stop[110] on
p_lu_data01_vol01 for client 3073: pid 18902 exited with return code 0
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_lu_data01_vol01_stop_0 (call=110, rc=0, cib-update=145,
confirmed=true) ok
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 23: stop p_target_data01_stop_0 on Storage1001B.xdomain.com (local)
Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: cancel_op: operation
monitor[103] on p_target_data01 for client 3073, its parameters:
crm_feature_set=[3.0.6] CRM_meta_timeout=[20000] CRM_meta_name=[monitor]
CRM_meta_interval=[10000] implementation=[lio]
iqn=[iqn.2013-10.com.xdomain.storage1001.data01] cancelled
Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: rsc:p_target_data01
stop[111] (pid 18952)
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 22: stop p_target_data01_stop_0 on Storage1001A.xdomain.com
Dec 17 14:05:50 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_target_data01_monitor_10000 (call=103, status=1, cib-update=0,
confirmed=true) Cancelled
Dec 17 14:05:50 Storage1001B kernel: [ 756.825198] BUG: unable to handle
kernel NULL pointer dereference at 0000000000000088
Dec 17 14:05:50 Storage1001B kernel: [ 756.825358] IP: [<ffffffffa04bf993>]
transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]
Dec 17 14:05:50 Storage1001B kernel: [ 756.825515] PGD 0
Dec 17 14:05:50 Storage1001B kernel: [ 756.825561] Oops: 0000 [#1] SMP
Dec 17 14:05:50 Storage1001B kernel: [ 756.825641] CPU 0
Dec 17 14:05:50 Storage1001B kernel: [ 756.825680] Modules linked in:
crc32c drbd lru_cache tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt
iscsi_target_mod target_core_pscsi target_core_file target_core_iblock
target_core_mod configfs nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc
bonding ext2 loop radeon ttm drm_kms_helper drm power_supply i2c_algo_bit
i5000_edac iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc snd_timer
processor edac_core thermal_sys i5k_amb snd soundcore i2c_i801 psmouse
joydev pcspkr serio_raw i2c_core coretemp evdev shpchp ioatdma rng_core dca
container button ext4 crc16 jbd2 mbcache sr_mod cdrom dm_mod usbhid hid sg
sd_mod crc_t10dif usb_storage uhci_hcd ahci libahci libata ehci_hcd usbcore
usb_common 3w_9xxx scsi_mod e1000e [last unloaded: scsi_wait_scan]
Dec 17 14:05:50 Storage1001B kernel: [ 756.827987]
Dec 17 14:05:50 Storage1001B kernel: [ 756.828025] Pid: 2903, comm:
iscsi_ttx Not tainted 3.2.0-4-amd64 #1 Debian 3.2.51-1 Supermicro
X7DBN/X7DBN
Dec 17 14:05:50 Storage1001B kernel: [ 756.828216] RIP:
0010:[<ffffffffa04bf993>] [<ffffffffa04bf993>]
transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]
Dec 17 14:05:50 Storage1001B kernel: [ 756.828393] RSP:
0018:ffff8801a7015d90 EFLAGS: 00010246
Dec 17 14:05:50 Storage1001B kernel: [ 756.828476] RAX: 0000000000000000
RBX: ffff8801a521e280 RCX: ffff8801a521e468
Dec 17 14:05:50 Storage1001B kernel: [ 756.828579] RDX: dead000000200200
RSI: 0000000000000286 RDI: ffff8801a7a9ab80
Dec 17 14:05:50 Storage1001B kernel: [ 756.828683] RBP: ffff8801a7a9abb8
R08: ffff8801a7015d90 R09: ffff8801a521e4a0
Dec 17 14:05:50 Storage1001B kernel: [ 756.828787] R10: ffff8801a4f39c00
R11: ffff8801a4f39c00 R12: ffff8801a7015d90
Dec 17 14:05:50 Storage1001B kernel: [ 756.828891] R13: ffff8801a7a9ab80
R14: 0000000000000008 R15: ffff8801a4f39d48
Dec 17 14:05:50 Storage1001B kernel: [ 756.828995] FS:
0000000000000000(0000) GS:ffff8801afc00000(0000) knlGS:0000000000000000
Dec 17 14:05:50 Storage1001B kernel: [ 756.829118] CS: 0010 DS: 0000 ES:
0000 CR0: 000000008005003b
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] CR2: 0000000000000088
CR3: 00000001a5620000 CR4: 00000000000006f0
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] Process iscsi_ttx (pid:
2903, threadinfo ffff8801a7014000, task ffff8801a5e93590)
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] Stack:
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] ffff8801a7015d90
ffff8801a7015d90 ffff880199232668 ffff8801a521e280
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] ffff8801a561a400
ffff8801a4f39ee8 ffff8801a4f39d48 ffffffffa04c0d22
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] ffff8801a4f39c00
ffffffffa04eef03 ffff8801a7015fd8 ffffffff81070fc1
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] Call Trace:
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffffa04c0d22>] ?
transport_generic_free_cmd+0x5c/0x65 [target_core_mod]
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffffa04eef03>] ?
iscsit_close_connection+0x120/0x4bb [iscsi_target_mod]
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff81070fc1>] ?
arch_local_irq_save+0x11/0x17
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffffa04f038a>] ?
iscsi_target_tx_thread+0x10ec/0x112b [iscsi_target_mod]
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff81039ac2>] ?
finish_task_switch+0x88/0xb9
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff8105fc83>] ?
add_wait_queue+0x3c/0x3c
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffffa04ef29e>] ?
iscsit_close_connection+0x4bb/0x4bb [iscsi_target_mod]
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff8105f631>] ?
kthread+0x76/0x7e
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff81356374>] ?
kernel_thread_helper+0x4/0x10
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff8105f5bb>] ?
kthread_worker_fn+0x139/0x139
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff81356370>] ?
gs_change+0x13/0x13
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] Code: 00 4c 8d 6d c8 74
0e 48 3b bb 10 02 00 00 74 05 e8 aa be c2 e0 48 89 ef e8 a7 f2 ff ff 48 8b
43 78 4c 89 ef 48 8b 80 90 02 00 00 <ff> 90 88 00 00 00 48 8b 2c 24 4c 39 e5
75 bc 48 83 c4 18 5b 5d
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] RIP
[<ffffffffa04bf993>] transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] RSP <ffff8801a7015d90>
Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] CR2: 0000000000000088
Dec 17 14:05:50 Storage1001B kernel: [ 756.864099] ---[ end trace
bbe9d829bd42ef9a ]---
Dec 17 14:05:51 Storage1001B kernel: [ 757.309687] BUG: unable to handle
kernel NULL pointer dereference at 0000000000000088
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] IP: [<ffffffffa04bf993>]
transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] PGD 1a6f1e067 PUD
1a6ffb067 PMD 0
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Oops: 0000 [#2] SMP
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] CPU 1
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Modules linked in:
crc32c drbd lru_cache tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt
iscsi_target_mod target_core_pscsi target_core_file target_core_iblock
target_core_mod configfs nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc
bonding ext2 loop radeon ttm drm_kms_helper drm power_supply i2c_algo_bit
i5000_edac iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc snd_timer
processor edac_core thermal_sys i5k_amb snd soundcore i2c_i801 psmouse
joydev pcspkr serio_raw i2c_core coretemp evdev shpchp ioatdma rng_core dca
container button ext4 crc16 jbd2 mbcache sr_mod cdrom dm_mod usbhid hid sg
sd_mod crc_t10dif usb_storage uhci_hcd ahci libahci libata ehci_hcd usbcore
usb_common 3w_9xxx scsi_mod e1000e [last unloaded: scsi_wait_scan]
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623]
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Pid: 2905, comm:
iscsi_ttx Tainted: G D 3.2.0-4-amd64 #1 Debian 3.2.51-1 Supermicro
X7DBN/X7DBN
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RIP:
0010:[<ffffffffa04bf993>] [<ffffffffa04bf993>]
transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RSP:
0018:ffff8801a74cfdf0 EFLAGS: 00010246
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RAX: 0000000000000000
RBX: ffff88019a161d00 RCX: ffff88019a161ee8
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RDX: dead000000200200
RSI: 0000000000000286 RDI: ffff8801a523ca80
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RBP: ffff8801a523cab8
R08: ffff8801a74cfdf0 R09: ffff88019a161f20
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] R10: ffff8801a576d438
R11: ffff8801a576d438 R12: ffff8801a74cfdf0
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] R13: ffff8801a523ca80
R14: 0000000000000000 R15: 0000000000000000
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] FS:
0000000000000000(0000) GS:ffff8801afc40000(0000) knlGS:0000000000000000
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] CS: 0010 DS: 0000 ES:
0000 CR0: 000000008005003b
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] CR2: 0000000000000088
CR3: 00000001a686d000 CR4: 00000000000006e0
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Process iscsi_ttx (pid:
2905, threadinfo ffff8801a74ce000, task ffff8801a5f68730)
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Stack:
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] ffff8801a74cfdf0
ffff8801a74cfdf0 ffff8801a6d7f668 ffff88019a161d00
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] ffff880199ccfc00
ffff88019a161ac0 ffff88019a161b29 ffffffffa04c0d22
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] ffff88019a161ba8
ffffffffa04ef4db ffff8801a5f68730 ffffffff81039ac2
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Call Trace:
Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] [<ffffffffa04c0d22>] ?
transport_generic_free_cmd+0x5c/0x65 [target_core_mod]
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffffa04ef4db>] ?
iscsi_target_tx_thread+0x23d/0x112b [iscsi_target_mod]
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff81039ac2>] ?
finish_task_switch+0x88/0xb9
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff8105fc83>] ?
add_wait_queue+0x3c/0x3c
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffffa04ef29e>] ?
iscsit_close_connection+0x4bb/0x4bb [iscsi_target_mod]
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff8105f631>] ?
kthread+0x76/0x7e
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff81356374>] ?
kernel_thread_helper+0x4/0x10
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff8105f5bb>] ?
kthread_worker_fn+0x139/0x139
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff81356370>] ?
gs_change+0x13/0x13
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] Code: 00 4c 8d 6d c8 74
0e 48 3b bb 10 02 00 00 74 05 e8 aa be c2 e0 48 89 ef e8 a7 f2 ff ff 48 8b
43 78 4c 89 ef 48 8b 80 90 02 00 00 <ff> 90 88 00 00 00 48 8b 2c 24 4c 39 e5
75 bc 48 83 c4 18 5b 5d
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] RIP
[<ffffffffa04bf993>] transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] RSP <ffff8801a74cfdf0>
Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] CR2: 0000000000000088
Dec 17 14:05:51 Storage1001B kernel: [ 757.593194] ---[ end trace
bbe9d829bd42ef9b ]---
Dec 17 14:05:53 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 111: monitor p_ping:0_monitor_5000 on Storage1001A.xdomain.com
Dec 17 14:05:58 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair,
id=status-Storage1001A.xdomain.com-p_ping, name=p_ping, value=200, magic=NA,
cib=0.264.47) : Transient attribute: update
Dec 17 14:06:00 Storage1001B kernel: [ 766.314376] iSCSI Login negotiation
failed.
Dec 17 14:06:00 Storage1001B kernel: [ 766.320550] iSCSI Login negotiation
failed.
Dec 17 14:06:03 Storage1001B kernel: [ 769.323055] iSCSI Login negotiation
failed.
Dec 17 14:06:03 Storage1001B kernel: [ 769.328979] iSCSI Login negotiation
failed.
Dec 17 14:06:04 Storage1001B kernel: [ 770.080981] iSCSI Login negotiation
failed.
Dec 17 14:06:04 Storage1001B kernel: [ 770.086770] iSCSI Login negotiation
failed.
Dec 17 14:06:06 Storage1001B kernel: [ 772.338555] iSCSI Login negotiation
failed.
Dec 17 14:06:06 Storage1001B kernel: [ 772.344101] iSCSI Login negotiation
failed.
Dec 17 14:06:07 Storage1001B kernel: [ 773.847066] iSCSI Login negotiation
failed.
Dec 17 14:06:07 Storage1001B kernel: [ 773.852207] iSCSI Login negotiation
failed.
Dec 17 14:06:10 Storage1001B kernel: [ 776.104111] iSCSI Login negotiation
failed.
Dec 17 14:06:10 Storage1001B kernel: [ 776.108920] iSCSI Login negotiation
failed.
Dec 17 14:06:10 Storage1001B lrmd: [3070]: WARN: p_target_data01:stop
process (PID 18952) timed out (try 1). Killing with signal SIGTERM (15).
Dec 17 14:06:10 Storage1001B lrmd: [3070]: WARN: operation stop[111] on
p_target_data01 for client 3073: pid 18952 timed out
Dec 17 14:06:10 Storage1001B crmd: [3073]: ERROR: process_lrm_event: LRM
operation p_target_data01_stop_0 (111) Timed Out (timeout=20000ms)
Dec 17 14:06:10 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 23
(p_target_data01_stop_0) on Storage1001B.xdomain.com failed (target: 0 vs.
rc: -2): Error
Dec 17 14:06:10 Storage1001B crmd: [3073]: WARN: update_failcount: Updating
failcount for p_target_data01 on Storage1001B.xdomain.com after failed stop:
rc=-2 (update=INFINITY, time=1387317970)
Dec 17 14:06:10 Storage1001B crmd: [3073]: info: abort_transition_graph:
match_graph_event:277 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=p_target_data01_last_failure_0,
magic=2:-2;23:4:0:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.48) :
Event failed
Dec 17 14:06:10 Storage1001B crmd: [3073]: notice: run_graph: ====
Transition 4 (Complete=13, Pending=0, Fired=0, Skipped=9, Incomplete=0,
Source=/var/lib/pengine/pe-error-0.bz2): Stopped
Dec 17 14:06:10 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 17 14:06:10 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: fail-count-p_target_data01 (INFINITY)
Dec 17 14:06:10 Storage1001B attrd: [3071]: notice: attrd_perform_update:
Sent update 45: fail-count-p_target_data01=INFINITY
Dec 17 14:06:10 Storage1001B attrd: [3071]: notice: attrd_trigger_update:
Sending flush op to all hosts for: last-failure-p_target_data01 (1387317970)
Dec 17 14:06:10 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair,
id=status-Storage1001B.xdomain.com-fail-count-p_target_data01,
name=fail-count-p_target_data01, value=INFINITY, magic=NA, cib=0.264.49) :
Transient attribute: update
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Dec 17 14:06:10 Storage1001B attrd: [3071]: notice: attrd_perform_update:
Sent update 48: last-failure-p_target_data01=1387317970
Dec 17 14:06:10 Storage1001B pengine: [3072]: WARN: unpack_rsc_op:
Processing failed op p_target_data01_last_failure_0 on
Storage1001B.xdomain.com: unknown exec error (-2)
Dec 17 14:06:10 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node
Storage1001B.xdomain.com will be fenced to recover from resource failure(s)
Dec 17 14:06:10 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node
Storage1001B.xdomain.com for STONITH
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice:
native_stop_constraints: Stop of failed resource p_target_data01 is implicit
after Storage1001B.xdomain.com is fenced
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Stop
st_node_a#011(Storage1001B.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Move
p_lvm01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip1#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip11#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Move
p_target_data01#011(Started Storage1001B.xdomain.com ->
Storage1001A.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Start
p_lu_data01_vol01#011(Storage1001A.xdomain.com)
Dec 17 14:06:10 Storage1001B crmd: [3073]: info: abort_transition_graph:
te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair,
id=status-Storage1001B.xdomain.com-last-failure-p_target_data01,
name=last-failure-p_target_data01, value=1387317970, magic=NA, cib=0.264.50)
: Transient attribute: update
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Start
p_email_admin1#011(Storage1001A.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd1:0#011(Slave -> Master Storage1001A.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Demote
p_drbd1:1#011(Master -> Stopped Storage1001B.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd2:0#011(Slave -> Master Storage1001A.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Demote
p_drbd2:1#011(Master -> Stopped Storage1001B.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_target:1#011(Storage1001B.xdomain.com)
Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_ping:1#011(Storage1001B.xdomain.com)
Dec 17 14:06:10 Storage1001B crmd: [3073]: info: handle_response: pe_calc
calculation pe_calc-dc-1387317970-109 is obsolete
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: process_pe_message:
Transition 5: WARNINGs found during PE processing. PEngine Input stored in:
/var/lib/pengine/pe-warn-180.bz2
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: unpack_rsc_op:
Processing failed op p_target_data01_last_failure_0 on
Storage1001B.xdomain.com: unknown exec error (-2)
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node
Storage1001B.xdomain.com will be fenced to recover from resource failure(s)
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: common_apply_stickiness:
Forcing p_target_data01 away from Storage1001B.xdomain.com after 1000000
failures (max=1000000)
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node
Storage1001B.xdomain.com for STONITH
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice:
native_stop_constraints: Stop of failed resource p_target_data01 is implicit
after Storage1001B.xdomain.com is fenced
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop
st_node_a#011(Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move
p_lvm01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip1#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip11#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move
p_target_data01#011(Started Storage1001B.xdomain.com ->
Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Start
p_lu_data01_vol01#011(Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Start
p_email_admin1#011(Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd1:0#011(Slave -> Master Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Demote
p_drbd1:1#011(Master -> Stopped Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd2:0#011(Slave -> Master Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Demote
p_drbd2:1#011(Master -> Stopped Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_target:1#011(Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_ping:1#011(Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: handle_response: pe_calc
calculation pe_calc-dc-1387317970-110 is obsolete
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: process_pe_message:
Transition 6: WARNINGs found during PE processing. PEngine Input stored in:
/var/lib/pengine/pe-warn-181.bz2
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: unpack_rsc_op:
Processing failed op p_target_data01_last_failure_0 on
Storage1001B.xdomain.com: unknown exec error (-2)
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node
Storage1001B.xdomain.com will be fenced to recover from resource failure(s)
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: common_apply_stickiness:
Forcing p_target_data01 away from Storage1001B.xdomain.com after 1000000
failures (max=1000000)
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node
Storage1001B.xdomain.com for STONITH
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice:
native_stop_constraints: Stop of failed resource p_target_data01 is implicit
after Storage1001B.xdomain.com is fenced
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop
st_node_a#011(Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move
p_lvm01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip1#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip11#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move
p_target_data01#011(Started Storage1001B.xdomain.com ->
Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Start
p_lu_data01_vol01#011(Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Start
p_email_admin1#011(Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd1:0#011(Slave -> Master Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Demote
p_drbd1:1#011(Master -> Stopped Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd2:0#011(Slave -> Master Storage1001A.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Demote
p_drbd2:1#011(Master -> Stopped Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_target:1#011(Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_ping:1#011(Storage1001B.xdomain.com)
Dec 17 14:06:11 Storage1001B kernel: [ 777.611642] iSCSI Login negotiation
failed.
Dec 17 14:06:11 Storage1001B kernel: [ 777.616045] iSCSI Login negotiation
failed.
Dec 17 14:06:11 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: do_te_invoke: Processing
graph 7 (ref=pe_calc-dc-1387317970-111) derived from
/var/lib/pengine/pe-warn-182.bz2
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 1: cancel p_drbd1:0_monitor_4000 on Storage1001A.xdomain.com
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 3: cancel p_drbd2:0_monitor_4000 on Storage1001A.xdomain.com
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 145: notify p_drbd1:0_pre_notify_demote_0 on Storage1001A.xdomain.com
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 147: notify p_drbd1:1_pre_notify_demote_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:06:11 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[112]
(pid 19574)
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 155: notify p_drbd2:0_pre_notify_demote_0 on Storage1001A.xdomain.com
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 157: notify p_drbd2:1_pre_notify_demote_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:06:11 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[113]
(pid 19575)
Dec 17 14:06:11 Storage1001B crmd: [3073]: notice: te_fence_node: Executing
reboot fencing operation (118) on Storage1001B.xdomain.com (timeout=60000)
Dec 17 14:06:11 Storage1001B stonith-ng: [3069]: info:
initiate_remote_stonith_op: Initiating remote operation reboot for
Storage1001B.xdomain.com: 832bbb47-e241-4702-bc48-dcd387841ed7
Dec 17 14:06:11 Storage1001B stonith-ng: [3069]: info:
can_fence_host_with_device: st_node_a can not fence
Storage1001B.xdomain.com: static-list
Dec 17 14:06:11 Storage1001B stonith-ng: [3069]: info: stonith_command:
Processed st_query from Storage1001B.xdomain.com: rc=0
Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: process_pe_message:
Transition 7: WARNINGs found during PE processing. PEngine Input stored in:
/var/lib/pengine/pe-warn-182.bz2
Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 17 14:06:11 Storage1001B lrmd: [3070]: info: operation notify[112] on
p_drbd1:1 for client 3073: pid 19574 exited with return code 0
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_notify_0 (call=112, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:06:11 Storage1001B lrmd: [3070]: info: operation notify[113] on
p_drbd2:1 for client 3073: pid 19575 exited with return code 0
Dec 17 14:06:11 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd2:1_notify_0 (call=113, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:06:13 Storage1001B kernel: [ 779.868256] iSCSI Login negotiation
failed.
Dec 17 14:06:13 Storage1001B kernel: [ 779.872318] iSCSI Login negotiation
failed.
Dec 17 14:06:15 Storage1001B kernel: [ 781.374749] iSCSI Login negotiation
failed.
Dec 17 14:06:15 Storage1001B kernel: [ 781.378561] iSCSI Login negotiation
failed.
Dec 17 14:06:17 Storage1001B stonith-ng: [3069]: ERROR: remote_op_done:
Operation reboot of Storage1001B.xdomain.com by <no-one> for
Storage1001B.xdomain.com[06b688b5-7fbe-4e14-8f63-6705013bd929]: Operation
timed out
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: tengine_stonith_callback:
StonithOp <remote-op state="0" st_target="Storage1001B.xdomain.com"
st_op="reboot" />
Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: tengine_stonith_callback:
Stonith operation 3 for Storage1001B.xdomain.com failed (Operation timed
out): aborting transition.
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: abort_transition_graph:
tengine_stonith_callback:454 - Triggered transition abort (complete=0) :
Stonith failed
Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: tengine_stonith_notify:
Peer Storage1001B.xdomain.com was not terminated (reboot) by <anyone> for
Storage1001B.xdomain.com: Operation timed out
(ref=832bbb47-e241-4702-bc48-dcd387841ed7)
Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: run_graph: ====
Transition 7 (Complete=16, Pending=0, Fired=0, Skipped=53, Incomplete=30,
Source=/var/lib/pengine/pe-warn-182.bz2): Stopped
Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 17 14:06:17 Storage1001B kernel: [ 783.630944] iSCSI Login negotiation
failed.
Dec 17 14:06:17 Storage1001B kernel: [ 783.634589] iSCSI Login negotiation
failed.
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Dec 17 14:06:17 Storage1001B pengine: [3072]: WARN: unpack_rsc_op:
Processing failed op p_target_data01_last_failure_0 on
Storage1001B.xdomain.com: unknown exec error (-2)
Dec 17 14:06:17 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node
Storage1001B.xdomain.com will be fenced to recover from resource failure(s)
Dec 17 14:06:17 Storage1001B pengine: [3072]: WARN: common_apply_stickiness:
Forcing p_target_data01 away from Storage1001B.xdomain.com after 1000000
failures (max=1000000)
Dec 17 14:06:17 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node
Storage1001B.xdomain.com for STONITH
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice:
native_stop_constraints: Stop of failed resource p_target_data01 is implicit
after Storage1001B.xdomain.com is fenced
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Stop
st_node_a#011(Storage1001B.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Move
p_lvm01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip1#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Move
p_ip11#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Move
p_target_data01#011(Started Storage1001B.xdomain.com ->
Storage1001A.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Start
p_lu_data01_vol01#011(Storage1001A.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Start
p_email_admin1#011(Storage1001A.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd1:0#011(Slave -> Master Storage1001A.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Demote
p_drbd1:1#011(Master -> Stopped Storage1001B.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Promote
p_drbd2:0#011(Slave -> Master Storage1001A.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Demote
p_drbd2:1#011(Master -> Stopped Storage1001B.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_target:1#011(Storage1001B.xdomain.com)
Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Stop
p_ping:1#011(Storage1001B.xdomain.com)
Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: do_te_invoke: Processing
graph 8 (ref=pe_calc-dc-1387317977-120) derived from
/var/lib/pengine/pe-warn-183.bz2
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 1: cancel p_drbd1:0_monitor_4000 on Storage1001A.xdomain.com
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 3: cancel p_drbd2:0_monitor_4000 on Storage1001A.xdomain.com
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 145: notify p_drbd1:0_pre_notify_demote_0 on Storage1001A.xdomain.com
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 147: notify p_drbd1:1_pre_notify_demote_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:06:17 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[114]
(pid 19824)
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 155: notify p_drbd2:0_pre_notify_demote_0 on Storage1001A.xdomain.com
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating
action 157: notify p_drbd2:1_pre_notify_demote_0 on Storage1001B.xdomain.com
(local)
Dec 17 14:06:17 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[115]
(pid 19825)
Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: te_fence_node: Executing
reboot fencing operation (118) on Storage1001B.xdomain.com (timeout=60000)
Dec 17 14:06:17 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001A.xdomain.com']//lrm_resource[@id='p_drbd1:0
']/lrm_rsc_op[@id='p_drbd1:0_monitor_4000']
(origin=Storage1001A.xdomain.com/crmd/33, version=0.264.53): ok (rc=0)
Dec 17 14:06:17 Storage1001B stonith-ng: [3069]: info:
initiate_remote_stonith_op: Initiating remote operation reboot for
Storage1001B.xdomain.com: cf2bcd82-7fe0-4316-8318-27e91a137adc
Dec 17 14:06:17 Storage1001B stonith-ng: [3069]: info:
can_fence_host_with_device: st_node_a can not fence
Storage1001B.xdomain.com: static-list
Dec 17 14:06:17 Storage1001B stonith-ng: [3069]: info: stonith_command:
Processed st_query from Storage1001B.xdomain.com: rc=0
Dec 17 14:06:17 Storage1001B cib: [3068]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='Storage1001A.xdomain.com']//lrm_resource[@id='p_drbd2:0
']/lrm_rsc_op[@id='p_drbd2:0_monitor_4000']
(origin=Storage1001A.xdomain.com/crmd/34, version=0.264.54): ok (rc=0)
Dec 17 14:06:17 Storage1001B lrmd: [3070]: info: operation notify[114] on
p_drbd1:1 for client 3073: pid 19824 exited with return code 0
Dec 17 14:06:17 Storage1001B lrmd: [3070]: info: operation notify[115] on
p_drbd2:1 for client 3073: pid 19825 exited with return code 0
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd1:1_notify_0 (call=114, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:06:17 Storage1001B crmd: [3073]: info: process_lrm_event: LRM
operation p_drbd2:1_notify_0 (call=115, rc=0, cib-update=0, confirmed=true)
ok
Dec 17 14:06:18 Storage1001B pengine: [3072]: WARN: process_pe_message:
Transition 8: WARNINGs found during PE processing. PEngine Input stored in:
/var/lib/pengine/pe-warn-183.bz2
Dec 17 14:06:18 Storage1001B pengine: [3072]: notice: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 17 14:06:19 Storage1001B kernel: [ 785.136801] iSCSI Login negotiation
failed.
Dec 17 14:06:19 Storage1001B kernel: [ 785.140293] iSCSI Login negotiation
failed.
Dec 17 14:06:21 Storage1001B kernel: [ 787.392813] iSCSI Login negotiation
failed.
Dec 17 14:06:21 Storage1001B kernel: [ 787.396060] iSCSI Login negotiation
failed.
Dec 17 14:06:22 Storage1001B kernel: [ 788.898172] iSCSI Login negotiation
failed.
Dec 17 14:06:22 Storage1001B kernel: [ 788.901280] iSCSI Login negotiation
failed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20131219/452bb592/attachment-0003.html>
More information about the Pacemaker
mailing list