[Pacemaker] pacemaker 1.1.10 + RHEL 7 beta issues

Digimer lists at alteeve.ca
Wed Jan 1 23:30:36 UTC 2014


I have a problem where putting a node into standby causes my "WebSite" 
resource to stop and refuse to start. This is the reproducer I show 
here, but I have seen this happen with DRBD earlier (before I realized 
what was wrong, so I didn't document it), so I don't think this is an 
apache RA issue.

[root at an-c03n01 ~]# rpm -q pacemaker httpd pcs corosync
pacemaker-1.1.10-19.el7.x86_64
httpd-2.4.6-7.el7.x86_64
pcs-0.9.99-2.el7.x86_64
corosync-2.3.2-2.el7.x86_64

[root at an-c03n01 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Everything release 7.0 Beta (Maipo)

[root at an-c03n01 ~]# uname -a
Linux an-c03n01.alteeve.ca 3.10.0-54.0.1.el7.x86_64 #1 SMP Tue Nov 26 
16:51:22 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

DRBD installed from source:

[root at an-c03n01 ~]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by 
root at an-c03n01.alteeve.ca, 2013-12-23 20:36:27


Starting with everything working:

=====
[root at an-c03n01 ~]# pcs status
Cluster name: an-cluster-03
Last updated: Wed Jan  1 18:13:19 2014
Last change: Wed Jan  1 18:03:08 2014 via cibadmin on an-c03n01.alteeve.ca
Stack: corosync
Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
7 Resources configured


Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]

Full list of resources:

  fence_n01_virsh	(stonith:fence_virsh):	Started an-c03n02.alteeve.ca
  fence_n02_virsh	(stonith:fence_virsh):	Started an-c03n02.alteeve.ca
  ClusterIP	(ocf::heartbeat:IPaddr2):	Started an-c03n01.alteeve.ca
  Master/Slave Set: WebDataClone [WebData]
      Masters: [ an-c03n01.alteeve.ca ]
      Slaves: [ an-c03n02.alteeve.ca ]
  WebFS	(ocf::heartbeat:Filesystem):	Started an-c03n01.alteeve.ca
  WebSite	(ocf::heartbeat:apache):	Started an-c03n01.alteeve.ca

PCSD Status:
an-c03n01.alteeve.ca:
   an-c03n01.alteeve.ca: Online
an-c03n02.alteeve.ca:
   an-c03n02.alteeve.ca: Online

Daemon Status:
   corosync: active/disabled
   pacemaker: active/disabled
   pcsd: active/enabled
=====

=====
[root at an-c03n01 ~]# pcs config show
Cluster Name: an-cluster-03
Corosync Nodes:
  an-c03n01.alteeve.ca an-c03n02.alteeve.ca
Pacemaker Nodes:
  an-c03n01.alteeve.ca an-c03n02.alteeve.ca

Resources:
  Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: ip=192.168.122.10 cidr_netmask=32
   Operations: monitor interval=30s (ClusterIP-monitor-interval-30s)
  Master: WebDataClone
   Meta Attrs: master-max=1 master-node-max=1 clone-max=2 
clone-node-max=1 notify=true
   Resource: WebData (class=ocf provider=linbit type=drbd)
    Attributes: drbd_resource=r0
    Operations: monitor interval=60s (WebData-monitor-60s)
  Resource: WebFS (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/drbd0 directory=/var/www/html fstype=ext4
   Operations: monitor interval=60s (WebFS-monitor-interval-60s)
  Resource: WebSite (class=ocf provider=heartbeat type=apache)
   Attributes: configfile=/etc/httpd/conf/httpd.conf 
statusurl=http://127.0.0.1/server-status
   Operations: monitor interval=60s (WebSite-monitor-interval-60s)

Stonith Devices:
  Resource: fence_n01_virsh (class=stonith type=fence_virsh)
   Attributes: pcmk_host_list=an-c03n01.alteeve.ca ipaddr=lemass 
login=root passwd_script=/root/lemass.pw delay=15 port=an-c03n01
   Operations: monitor interval=60s (fence_n01_virsh-monitor-interval-60s)
  Resource: fence_n02_virsh (class=stonith type=fence_virsh)
   Attributes: pcmk_host_list=an-c03n02.alteeve.ca ipaddr=lemass 
login=root passwd_script=/root/lemass.pw port=an-c03n02
   Operations: monitor interval=60s (fence_n02_virsh-monitor-interval-60s)
Fencing Levels:

Location Constraints:
   Resource: ClusterIP
     Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started) 
(id:cli-prefer-ClusterIP)
   Resource: WebDataClone
     Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started) 
(id:cli-prefer-WebDataClone)
   Resource: WebSite
     Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started) 
(id:cli-prefer-WebSite)
Ordering Constraints:
   promote WebDataClone then start WebFS (Mandatory) 
(id:order-WebDataClone-WebFS-mandatory)
   start WebFS then start WebSite (Mandatory) 
(id:order-WebFS-WebSite-mandatory)
Colocation Constraints:
   WebFS with WebDataClone (INFINITY) (with-rsc-role:Master) 
(id:colocation-WebFS-WebDataClone-INFINITY)
   WebSite with ClusterIP (INFINITY) 
(id:colocation-WebSite-ClusterIP-INFINITY)
   WebSite with WebFS (INFINITY) (id:colocation-WebSite-WebFS-INFINITY)

Cluster Properties:
  cluster-infrastructure: corosync
  dc-version: 1.1.10-19.el7-368c726
  last-lrm-refresh: 1388615115
  no-quorum-policy: ignore
  stonith-enabled: true
=====

Put an-c03n01 into standby:

=====
pcs cluster standby an-c03n01.alteeve.ca

==[an-c03n01 logs]===
Jan  1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.189.16
Jan  1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.190.1 
87f59f171d277d9cb1b51cbda41fbf24
Jan  1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: -- <cib 
admin_epoch="0" epoch="189" num_updates="16"/>
Jan  1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++ 
<instance_attributes id="nodes-1">
Jan  1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++ 
<nvpair id="nodes-1-standby" name="standby" value="on"/>
Jan  1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++ 
</instance_attributes>
Jan  1 18:16:31 an-c03n01 crmd[1767]: notice: do_state_transition: State 
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC 
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jan  1 18:16:31 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move 
ClusterIP	(Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
Jan  1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Demote 
WebData:0	(Master -> Stopped an-c03n01.alteeve.ca)
Jan  1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Promote 
WebData:1	(Slave -> Master an-c03n02.alteeve.ca)
Jan  1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move 
WebFS	(Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
Jan  1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move 
WebSite	(Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
Jan  1 18:16:31 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 37: /var/lib/pacemaker/pengine/pe-input-176.bz2
Jan  1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 15: stop ClusterIP_stop_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 6: cancel WebData_cancel_60000 on an-c03n02.alteeve.ca
Jan  1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 50: stop WebSite_stop_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 64: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca 
(local)
Jan  1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 66: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
Jan  1 18:16:31 an-c03n01 apache(WebSite)[14743]: INFO: Attempting 
graceful stop of apache PID 10641
Jan  1 18:16:31 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=97, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:31 an-c03n01 IPaddr2(ClusterIP)[14742]: INFO: IP status = 
ok, IP_CIP=
Jan  1 18:16:31 an-c03n01 avahi-daemon[298]: Withdrawing address record 
for 192.168.122.10 on eth0.
Jan  1 18:16:31 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation ClusterIP_stop_0 (call=94, rc=0, cib-update=283, 
confirmed=true) ok
Jan  1 18:16:33 an-c03n01 apache(WebSite)[14743]: INFO: apache stopped.
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebSite_stop_0 (call=96, rc=0, cib-update=284, confirmed=true) ok
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: run_graph: Transition 37 
(Complete=6, Pending=0, Fired=0, Skipped=22, Incomplete=13, 
Source=/var/lib/pacemaker/pengine/pe-input-176.bz2): Stopped
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start 
ClusterIP	(an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Demote 
WebData:0	(Master -> Stopped an-c03n01.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote 
WebData:1	(Slave -> Master an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Move 
WebFS	(Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebSite	(an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 38: /var/lib/pacemaker/pengine/pe-input-177.bz2
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 12: start ClusterIP_start_0 on an-c03n02.alteeve.ca
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 43: stop WebFS_stop_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 59: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca 
(local)
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 61: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=100, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: Running stop 
for /dev/drbd0 on /var/www/html
Jan  1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: Trying to 
unmount /var/www/html
Jan  1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: unmounted 
/var/www/html successfully
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebFS_stop_0 (call=99, rc=0, cib-update=286, confirmed=true) ok
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 14: demote WebData_demote_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 13: monitor ClusterIP_monitor_30000 on an-c03n02.alteeve.ca
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.547981] block drbd0: role( 
Primary -> Secondary )
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.548034] block drbd0: bitmap 
WRITE of 0 pages took 0 jiffies
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.548037] block drbd0: 0 KB (0 
bits) marked out-of-sync by on disk bit-map.
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_demote_0 (call=101, rc=0, cib-update=287, 
confirmed=true) ok
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 60: notify WebData_post_notify_demote_0 on an-c03n01.alteeve.ca 
(local)
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 62: notify WebData_post_notify_demote_0 on an-c03n02.alteeve.ca
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=102, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 54: notify WebData_pre_notify_stop_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 55: notify WebData_pre_notify_stop_0 on an-c03n02.alteeve.ca
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=103, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 15: stop WebData_stop_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.638480] drbd r0: peer( 
Secondary -> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate 
-> DUnknown )
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.638490] drbd r0: asender terminated
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.638492] drbd r0: Terminating 
drbd_a_r0
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.638732] drbd r0: Connection closed
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.638741] drbd r0: conn( 
Disconnecting -> StandAlone )
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.638742] drbd r0: receiver 
terminated
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.638743] drbd r0: Terminating 
drbd_r_r0
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.638776] block drbd0: disk( 
UpToDate -> Failed )
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.651161] block drbd0: bitmap 
WRITE of 0 pages took 0 jiffies
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.651164] block drbd0: 0 KB (0 
bits) marked out-of-sync by on disk bit-map.
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.651167] block drbd0: disk( 
Failed -> Diskless )
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.651239] block drbd0: 
drbd_bm_resize called with capacity == 0
Jan  1 18:16:33 an-c03n01 kernel: [ 2911.651279] drbd r0: Terminating 
drbd_w_r0
Jan  1 18:16:33 an-c03n01 attrd[1765]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: master-WebData (<null>)
Jan  1 18:16:33 an-c03n01 attrd[1765]: notice: attrd_perform_update: 
Sent delete 154: node=1, attr=master-WebData, id=<n/a>, set=(null), 
section=status
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_stop_0 (call=104, rc=0, cib-update=288, confirmed=true) ok
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 56: notify WebData_post_notify_stop_0 on an-c03n02.alteeve.ca
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: run_graph: Transition 38 
(Complete=24, Pending=0, Fired=0, Skipped=10, Incomplete=4, 
Source=/var/lib/pacemaker/pengine/pe-input-177.bz2): Stopped
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote 
WebData:0	(Slave -> Master an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebFS	(an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebSite	(an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 39: /var/lib/pacemaker/pengine/pe-input-178.bz2
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote 
WebData:0	(Slave -> Master an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebFS	(an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebSite	(an-c03n02.alteeve.ca)
Jan  1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 40: /var/lib/pacemaker/pengine/pe-input-178.bz2
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 54: notify WebData_pre_notify_promote_0 on an-c03n02.alteeve.ca
Jan  1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 16: promote WebData_promote_0 on an-c03n02.alteeve.ca
Jan  1 18:16:34 an-c03n01 crmd[1767]: notice: handle_request: Current 
ping state: S_TRANSITION_ENGINE
Jan  1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.190.11
Jan  1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.191.1 
ba2e8658b93b742e3ef335fa93b50c26
Jan  1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: -- <cib 
admin_epoch="0" epoch="190" num_updates="11"/>
Jan  1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ 
<rsc_location rsc="WebDataClone" id="drbd-fence-by-handler-r0-WebDataClone">
Jan  1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++         <rule 
role="Master" score="-INFINITY" 
id="drbd-fence-by-handler-r0-rule-WebDataClone">
Jan  1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ 
<expression attribute="#uname" operation="ne" 
value="an-c03n02.alteeve.ca" 
id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
Jan  1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++         </rule>
Jan  1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ 
</rsc_location>
Jan  1 18:16:34 an-c03n01 stonith-ng[1763]: notice: unpack_config: On 
loss of CCM Quorum: Ignore
Jan  1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 55: notify WebData_post_notify_promote_0 on an-c03n02.alteeve.ca
Jan  1 18:16:34 an-c03n01 stonith-ng[1763]: notice: 
stonith_device_register: Added 'fence_n01_virsh' to the device list (2 
active devices)
Jan  1 18:16:34 an-c03n01 stonith-ng[1763]: notice: 
stonith_device_register: Added 'fence_n02_virsh' to the device list (2 
active devices)
Jan  1 18:16:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 40 
(Complete=9, Pending=0, Fired=0, Skipped=4, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-178.bz2): Stopped
Jan  1 18:16:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:16:34 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebFS	(an-c03n02.alteeve.ca)
Jan  1 18:16:34 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebSite	(an-c03n02.alteeve.ca)
Jan  1 18:16:34 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 41: /var/lib/pacemaker/pengine/pe-input-179.bz2
Jan  1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 42: start WebFS_start_0 on an-c03n02.alteeve.ca
Jan  1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 43: monitor WebFS_monitor_60000 on an-c03n02.alteeve.ca
Jan  1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 44: start WebSite_start_0 on an-c03n02.alteeve.ca
Jan  1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 45: monitor WebSite_monitor_60000 on an-c03n02.alteeve.ca
Jan  1 18:16:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 41 
(Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-179.bz2): Complete
Jan  1 18:16:34 an-c03n01 crmd[1767]: notice: do_state_transition: State 
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS 
cause=C_FSA_INTERNAL origin=notify_crmd ]

==[an-c03n02 logs]===
Jan  1 18:16:31 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=90, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=92, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO: Adding inet 
address 192.168.122.10/32 to device eth0
Jan  1 18:16:33 an-c03n02 avahi-daemon[295]: Registering new address 
record for 192.168.122.10 on eth0.IPv4.
Jan  1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO: Bringing 
device eth0 up
Jan  1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO: 
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p 
/var/run/resource-agents/send_arp-192.168.122.10 eth0 192.168.122.10 
auto not_used not_used
Jan  1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation ClusterIP_start_0 (call=91, rc=0, cib-update=62, 
confirmed=true) ok
Jan  1 18:16:33 an-c03n02 kernel: [19473.555292] block drbd0: peer( 
Primary -> Secondary )
Jan  1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation ClusterIP_monitor_30000 (call=93, rc=0, cib-update=63, 
confirmed=false) ok
Jan  1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=94, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=95, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:33 an-c03n02 kernel: [19473.645249] drbd r0: peer( 
Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> 
DUnknown )
Jan  1 18:16:33 an-c03n02 kernel: [19473.645267] drbd r0: asender terminated
Jan  1 18:16:33 an-c03n02 kernel: [19473.645269] drbd r0: Terminating 
drbd_a_r0
Jan  1 18:16:33 an-c03n02 kernel: [19473.645531] drbd r0: Connection closed
Jan  1 18:16:33 an-c03n02 kernel: [19473.645538] drbd r0: conn( TearDown 
-> Unconnected )
Jan  1 18:16:33 an-c03n02 kernel: [19473.645539] drbd r0: receiver 
terminated
Jan  1 18:16:33 an-c03n02 kernel: [19473.645540] drbd r0: Restarting 
receiver thread
Jan  1 18:16:33 an-c03n02 kernel: [19473.645541] drbd r0: receiver 
(re)started
Jan  1 18:16:33 an-c03n02 kernel: [19473.645546] drbd r0: conn( 
Unconnected -> WFConnection )
Jan  1 18:16:33 an-c03n02 attrd[16386]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: master-WebData (1000)
Jan  1 18:16:33 an-c03n02 attrd[16386]: notice: attrd_perform_update: 
Sent update 152: master-WebData=1000
Jan  1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=96, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=97, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:34 an-c03n02 kernel: [19473.791812] drbd r0: helper 
command: /sbin/drbdadm fence-peer r0
Jan  1 18:16:34 an-c03n02 crm-fence-peer.sh[32287]: invoked for r0
Jan  1 18:16:34 an-c03n02 cibadmin[32319]: notice: crm_log_args: 
Invoked: cibadmin -C -o constraints -X <rsc_location rsc="WebDataClone" 
id="drbd-fence-by-handler-r0-WebDataClone">
   <rule role="Master" score="-INFINITY" 
id="drbd-fence-by-handler-r0-rule-WebDataClone">
     <expression attribute="#uname" operation="ne" 
value="an-c03n02.alteeve.ca" 
id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
   </rule>
</rsc_location>
Jan  1 18:16:34 an-c03n02 kernel: [19473.836661] drbd r0: helper 
command: /sbin/drbdadm fence-peer r0 exit code 4 (0x400)
Jan  1 18:16:34 an-c03n02 kernel: [19473.836664] drbd r0: fence-peer 
helper returned 4 (peer was fenced)
Jan  1 18:16:34 an-c03n02 kernel: [19473.836676] drbd r0: pdsk( DUnknown 
-> Outdated )
Jan  1 18:16:34 an-c03n02 kernel: [19473.836681] block drbd0: role( 
Secondary -> Primary )
Jan  1 18:16:34 an-c03n02 stonith-ng[16384]: notice: unpack_config: On 
loss of CCM Quorum: Ignore
Jan  1 18:16:34 an-c03n02 crm-fence-peer.sh[32287]: INFO peer is 
reachable, my disk is UpToDate: placed constraint 
'drbd-fence-by-handler-r0-WebDataClone'
Jan  1 18:16:34 an-c03n02 kernel: [19473.837205] block drbd0: new 
current UUID 
3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC
Jan  1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_promote_0 (call=98, rc=0, cib-update=64, 
confirmed=true) ok
Jan  1 18:16:34 an-c03n02 stonith-ng[16384]: notice: 
stonith_device_register: Device 'fence_n01_virsh' already existed in 
device list (2 active devices)
Jan  1 18:16:34 an-c03n02 attrd[16386]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: master-WebData (10000)
Jan  1 18:16:34 an-c03n02 attrd[16386]: notice: attrd_perform_update: 
Sent update 154: master-WebData=10000
Jan  1 18:16:34 an-c03n02 stonith-ng[16384]: notice: 
stonith_device_register: Device 'fence_n02_virsh' already existed in 
device list (2 active devices)
Jan  1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=99, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:16:34 an-c03n02 Filesystem(WebFS)[32362]: INFO: Running start 
for /dev/drbd0 on /var/www/html
Jan  1 18:16:34 an-c03n02 kernel: [19473.948256] EXT4-fs (drbd0): 
mounted filesystem with ordered data mode. Opts: (null)
Jan  1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebFS_start_0 (call=100, rc=0, cib-update=65, confirmed=true) ok
Jan  1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebFS_monitor_60000 (call=101, rc=0, cib-update=66, 
confirmed=false) ok
Jan  1 18:16:34 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
Jan  1 18:16:34 an-c03n02 apache(WebSite)[32427]: INFO: AH00558: httpd: 
Could not reliably determine the server's fully qualified domain name, 
using an-c03n02.alteeve.ca. Set the 'ServerName' directive globally to 
suppress this message
Jan  1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebSite_start_0 (call=102, rc=0, cib-update=67, confirmed=true) ok
Jan  1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebSite_monitor_60000 (call=103, rc=0, cib-update=68, 
confirmed=false) ok
Jan  1 18:16:43 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
Jan  1 18:16:53 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
Jan  1 18:17:03 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
Jan  1 18:17:13 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
Jan  1 18:17:23 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
Jan  1 18:17:33 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
Jan  1 18:17:43 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
Jan  1 18:17:53 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
Jan  1 18:18:03 an-c03n02 systemd: pacemaker.service: Got notification 
message from PID 32497, but reception only permitted for PID 16382
=====

Not sure why this happened:
=====
Jan  1 18:16:34 an-c03n02 cibadmin[32319]: notice: crm_log_args: 
Invoked: cibadmin -C -o constraints -X <rsc_location rsc="WebDataClone" 
id="drbd-fence-by-handler-r0-WebDataClone">
   <rule role="Master" score="-INFINITY" 
id="drbd-fence-by-handler-r0-rule-WebDataClone">
     <expression attribute="#uname" operation="ne" 
value="an-c03n02.alteeve.ca" 
id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
   </rule>
</rsc_location>
=====

That seems to be the problem, but let me continue.

Everything fails over properly:

=====
[root at an-c03n01 ~]# pcs status
Cluster name: an-cluster-03
Last updated: Wed Jan  1 18:19:01 2014
Last change: Wed Jan  1 18:16:34 2014 via cibadmin on an-c03n02.alteeve.ca
Stack: corosync
Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
7 Resources configured


Node an-c03n01.alteeve.ca (1): standby
Online: [ an-c03n02.alteeve.ca ]

Full list of resources:

  fence_n01_virsh	(stonith:fence_virsh):	Started an-c03n02.alteeve.ca
  fence_n02_virsh	(stonith:fence_virsh):	Started an-c03n02.alteeve.ca
  ClusterIP	(ocf::heartbeat:IPaddr2):	Started an-c03n02.alteeve.ca
  Master/Slave Set: WebDataClone [WebData]
      Masters: [ an-c03n02.alteeve.ca ]
      Stopped: [ an-c03n01.alteeve.ca ]
  WebFS	(ocf::heartbeat:Filesystem):	Started an-c03n02.alteeve.ca
  WebSite	(ocf::heartbeat:apache):	Started an-c03n02.alteeve.ca

PCSD Status:
an-c03n01.alteeve.ca:
   an-c03n01.alteeve.ca: Online
an-c03n02.alteeve.ca:
   an-c03n02.alteeve.ca: Online

Daemon Status:
   corosync: active/disabled
   pacemaker: active/disabled
   pcsd: active/enabled
=====

Current config:
=====
[root at an-c03n01 ~]# pcs config show
Cluster Name: an-cluster-03
Corosync Nodes:
  an-c03n01.alteeve.ca an-c03n02.alteeve.ca
Pacemaker Nodes:
  an-c03n01.alteeve.ca an-c03n02.alteeve.ca

Resources:
  Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: ip=192.168.122.10 cidr_netmask=32
   Operations: monitor interval=30s (ClusterIP-monitor-interval-30s)
  Master: WebDataClone
   Meta Attrs: master-max=1 master-node-max=1 clone-max=2 
clone-node-max=1 notify=true
   Resource: WebData (class=ocf provider=linbit type=drbd)
    Attributes: drbd_resource=r0
    Operations: monitor interval=60s (WebData-monitor-60s)
  Resource: WebFS (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/drbd0 directory=/var/www/html fstype=ext4
   Operations: monitor interval=60s (WebFS-monitor-interval-60s)
  Resource: WebSite (class=ocf provider=heartbeat type=apache)
   Attributes: configfile=/etc/httpd/conf/httpd.conf 
statusurl=http://127.0.0.1/server-status
   Operations: monitor interval=60s (WebSite-monitor-interval-60s)

Stonith Devices:
  Resource: fence_n01_virsh (class=stonith type=fence_virsh)
   Attributes: pcmk_host_list=an-c03n01.alteeve.ca ipaddr=lemass 
login=root passwd_script=/root/lemass.pw delay=15 port=an-c03n01
   Operations: monitor interval=60s (fence_n01_virsh-monitor-interval-60s)
  Resource: fence_n02_virsh (class=stonith type=fence_virsh)
   Attributes: pcmk_host_list=an-c03n02.alteeve.ca ipaddr=lemass 
login=root passwd_script=/root/lemass.pw port=an-c03n02
   Operations: monitor interval=60s (fence_n02_virsh-monitor-interval-60s)
Fencing Levels:

Location Constraints:
   Resource: ClusterIP
     Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started) 
(id:cli-prefer-ClusterIP)
   Resource: WebDataClone
     Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started) 
(id:cli-prefer-WebDataClone)
     Constraint: drbd-fence-by-handler-r0-WebDataClone
       Rule: score=-INFINITY role=Master 
(id:drbd-fence-by-handler-r0-rule-WebDataClone)
         Expression: #uname ne an-c03n02.alteeve.ca 
(id:drbd-fence-by-handler-r0-expr-WebDataClone)
   Resource: WebSite
     Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started) 
(id:cli-prefer-WebSite)
Ordering Constraints:
   promote WebDataClone then start WebFS (Mandatory) 
(id:order-WebDataClone-WebFS-mandatory)
   start WebFS then start WebSite (Mandatory) 
(id:order-WebFS-WebSite-mandatory)
Colocation Constraints:
   WebFS with WebDataClone (INFINITY) (with-rsc-role:Master) 
(id:colocation-WebFS-WebDataClone-INFINITY)
   WebSite with ClusterIP (INFINITY) 
(id:colocation-WebSite-ClusterIP-INFINITY)
   WebSite with WebFS (INFINITY) (id:colocation-WebSite-WebFS-INFINITY)

Cluster Properties:
  cluster-infrastructure: corosync
  dc-version: 1.1.10-19.el7-368c726
  last-lrm-refresh: 1388615115
  no-quorum-policy: ignore
  stonith-enabled: true
Node Attributes:
  an-c03n01.alteeve.ca: standby=on
=====

Put an-c03n01 into unstandby:

=====
[root at an-c03n01 ~]# pcs cluster unstandby an-c03n01.alteeve.ca

==[an-c03n01 logs]===
Jan  1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.191.7
Jan  1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.192.1 
9eac78791868c0b3878e59927a934eba
Jan  1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: -- 
<nvpair id="nodes-1-standby" name="standby" value="on"/>
Jan  1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: ++ <cib 
admin_epoch="0" cib-last-written="Wed Jan  1 18:22:32 2014" 
crm_feature_set="3.0.7" epoch="192" have-quorum="1" num_updates="1" 
update-client="crm_attribute" update-origin="an-c03n01.alteeve.ca" 
validate-with="pacemaker-1.2" dc-uuid="1"/>
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: do_state_transition: State 
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC 
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jan  1 18:22:32 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Move 
ClusterIP	(Started an-c03n02.alteeve.ca -> an-c03n01.alteeve.ca)
Jan  1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebData:1	(an-c03n01.alteeve.ca)
Jan  1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Stop 
WebSite	(an-c03n02.alteeve.ca)
Jan  1 18:22:32 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 42: /var/lib/pacemaker/pengine/pe-input-180.bz2
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 14: stop ClusterIP_stop_0 on an-c03n02.alteeve.ca
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 49: stop WebSite_stop_0 on an-c03n02.alteeve.ca
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 53: notify WebData_pre_notify_start_0 on an-c03n02.alteeve.ca
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 21: start WebData:1_start_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 15: start ClusterIP_start_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO: Adding inet 
address 192.168.122.10/32 to device eth0
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.282453] drbd r0: Starting 
worker thread (from drbdsetup [15705])
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.283364] block drbd0: disk( 
Diskless -> Attaching )
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.283429] drbd r0: Method to 
ensure write ordering: drain
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.283431] block drbd0: max BIO 
size = 1048576
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.283433] block drbd0: Adjusting 
my ra_pages to backing device's (32 -> 1024)
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.283435] block drbd0: 
drbd_bm_resize called with capacity == 102392704
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.283580] block drbd0: resync 
bitmap: bits=12799088 words=199986 pages=391
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.283582] block drbd0: size = 49 
GB (51196352 KB)
Jan  1 18:22:32 an-c03n01 avahi-daemon[298]: Registering new address 
record for 192.168.122.10 on eth0.IPv4.
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.285244] block drbd0: bitmap 
READ of 391 pages took 2 jiffies
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.285466] block drbd0: recounting 
of set bits took additional 0 jiffies
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.285468] block drbd0: 0 KB (0 
bits) marked out-of-sync by on disk bit-map.
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.285473] block drbd0: disk( 
Attaching -> Consistent )
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.285475] block drbd0: attached 
to UUIDs 6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
Jan  1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO: Bringing 
device eth0 up
Jan  1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO: 
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p 
/var/run/resource-agents/send_arp-192.168.122.10 eth0 192.168.122.10 
auto not_used not_used
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.292172] drbd r0: conn( 
StandAlone -> Unconnected )
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.292185] drbd r0: Starting 
receiver thread (from drbd_w_r0 [15707])
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.292199] drbd r0: receiver 
(re)started
Jan  1 18:22:32 an-c03n01 kernel: [ 3270.292205] drbd r0: conn( 
Unconnected -> WFConnection )
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation ClusterIP_start_0 (call=106, rc=0, cib-update=293, 
confirmed=true) ok
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 16: monitor ClusterIP_monitor_30000 on an-c03n01.alteeve.ca (local)
Jan  1 18:22:32 an-c03n01 attrd[1765]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: master-WebData (5)
Jan  1 18:22:32 an-c03n01 attrd[1765]: notice: attrd_perform_update: 
Sent update 160: master-WebData=5
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_start_0 (call=105, rc=0, cib-update=294, 
confirmed=true) ok
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 54: notify WebData_post_notify_start_0 on an-c03n02.alteeve.ca
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 55: notify WebData:1_post_notify_start_0 on an-c03n01.alteeve.ca 
(local)
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation ClusterIP_monitor_30000 (call=107, rc=0, cib-update=295, 
confirmed=false) ok
Jan  1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=108, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.793761] drbd r0: Handshake 
successful: Agreed network protocol version 101
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.793765] drbd r0: Agreed to 
support TRIM on protocol level
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.793809] drbd r0: conn( 
WFConnection -> WFReportParams )
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.793812] drbd r0: Starting 
asender thread (from drbd_r_r0 [15727])
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.799253] block drbd0: 
drbd_sync_handshake:
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.799258] block drbd0: self 
6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC 
bits:0 flags:0
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.799261] block drbd0: peer 
3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC 
bits:1 flags:0
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.799263] block drbd0: 
uuid_compare()=-1 by rule 50
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.799270] block drbd0: peer( 
Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( 
Consistent -> Outdated ) pdsk( DUnknown -> UpToDate )
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.803153] block drbd0: receive 
bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; 
compression: 100.0%
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.803403] block drbd0: send 
bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; 
compression: 100.0%
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.803409] block drbd0: conn( 
WFBitMapT -> WFSyncUUID )
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.805842] block drbd0: updated 
sync uuid 
6A9A4DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.806189] block drbd0: helper 
command: /sbin/drbdadm before-resync-target minor-0
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.807379] block drbd0: helper 
command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.807392] block drbd0: conn( 
WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.807398] block drbd0: Began 
resync as SyncTarget (will sync 4 KB [1 bits set]).
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.809259] block drbd0: Resync 
done (total 1 sec; paused 0 sec; 4 K/sec)
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.809264] block drbd0: updated 
UUIDs 3E5C4A9426BEF114:0000000000000000:6A9A4DBF33FF45AE:6A994DBF33FF45AE
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.809268] block drbd0: conn( 
SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.809382] block drbd0: helper 
command: /sbin/drbdadm after-resync-target minor-0
Jan  1 18:22:33 an-c03n01 kernel: [ 3270.810777] block drbd0: helper 
command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)
Jan  1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 42 
(Complete=14, Pending=0, Fired=0, Skipped=2, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-180.bz2): Stopped
Jan  1 18:22:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:22:34 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 43: /var/lib/pacemaker/pengine/pe-input-181.bz2
Jan  1 18:22:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 17: monitor WebData_monitor_60000 on an-c03n01.alteeve.ca (local)
Jan  1 18:22:34 an-c03n01 attrd[1765]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: master-WebData (10000)
Jan  1 18:22:34 an-c03n01 attrd[1765]: notice: attrd_perform_update: 
Sent update 162: master-WebData=10000
Jan  1 18:22:34 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_monitor_60000 (call=109, rc=0, cib-update=297, 
confirmed=false) ok
Jan  1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 43 
(Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-181.bz2): Complete
Jan  1 18:22:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:22:34 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 44: /var/lib/pacemaker/pengine/pe-input-182.bz2
Jan  1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 44 
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-182.bz2): Complete
Jan  1 18:22:34 an-c03n01 crmd[1767]: notice: do_state_transition: State 
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS 
cause=C_FSA_INTERNAL origin=notify_crmd ]

==[an-c03n02 logs]===
Jan  1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=108, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:22:32 an-c03n02 apache(WebSite)[2631]: INFO: Attempting 
graceful stop of apache PID 32497
Jan  1 18:22:32 an-c03n02 IPaddr2(ClusterIP)[2630]: INFO: IP status = 
ok, IP_CIP=
Jan  1 18:22:32 an-c03n02 avahi-daemon[295]: Withdrawing address record 
for 192.168.122.10 on eth0.
Jan  1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation ClusterIP_stop_0 (call=105, rc=0, cib-update=69, 
confirmed=true) ok
Jan  1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=109, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:22:33 an-c03n02 kernel: [19832.801173] drbd r0: Handshake 
successful: Agreed network protocol version 101
Jan  1 18:22:33 an-c03n02 kernel: [19832.801177] drbd r0: Agreed to 
support TRIM on protocol level
Jan  1 18:22:33 an-c03n02 kernel: [19832.801209] drbd r0: conn( 
WFConnection -> WFReportParams )
Jan  1 18:22:33 an-c03n02 kernel: [19832.801211] drbd r0: Starting 
asender thread (from drbd_r_r0 [19965])
Jan  1 18:22:33 an-c03n02 kernel: [19832.810255] block drbd0: 
drbd_sync_handshake:
Jan  1 18:22:33 an-c03n02 kernel: [19832.810260] block drbd0: self 
3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC 
bits:1 flags:0
Jan  1 18:22:33 an-c03n02 kernel: [19832.810263] block drbd0: peer 
6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC 
bits:0 flags:0
Jan  1 18:22:33 an-c03n02 kernel: [19832.810265] block drbd0: 
uuid_compare()=1 by rule 70
Jan  1 18:22:33 an-c03n02 kernel: [19832.810270] block drbd0: peer( 
Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( 
Outdated -> Consistent )
Jan  1 18:22:33 an-c03n02 kernel: [19832.810573] block drbd0: send 
bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; 
compression: 100.0%
Jan  1 18:22:33 an-c03n02 kernel: [19832.811274] block drbd0: receive 
bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; 
compression: 100.0%
Jan  1 18:22:33 an-c03n02 kernel: [19832.811279] block drbd0: helper 
command: /sbin/drbdadm before-resync-source minor-0
Jan  1 18:22:33 an-c03n02 kernel: [19832.812937] block drbd0: helper 
command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
Jan  1 18:22:33 an-c03n02 kernel: [19832.812951] block drbd0: conn( 
WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
Jan  1 18:22:33 an-c03n02 kernel: [19832.812959] block drbd0: Began 
resync as SyncSource (will sync 4 KB [1 bits set]).
Jan  1 18:22:33 an-c03n02 kernel: [19832.812996] block drbd0: updated 
sync UUID 
3E5C4A9426BEF115:6A9A4DBF33FF45AE:6A994DBF33FF45AE:2A36647E5537B8FC
Jan  1 18:22:33 an-c03n02 kernel: [19832.816963] block drbd0: Resync 
done (total 1 sec; paused 0 sec; 4 K/sec)
Jan  1 18:22:33 an-c03n02 kernel: [19832.816978] block drbd0: updated 
UUIDs 3E5C4A9426BEF115:0000000000000000:6A9A4DBF33FF45AE:6A994DBF33FF45AE
Jan  1 18:22:33 an-c03n02 kernel: [19832.816985] block drbd0: conn( 
SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
Jan  1 18:22:34 an-c03n02 apache(WebSite)[2631]: INFO: apache stopped.
Jan  1 18:22:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebSite_stop_0 (call=107, rc=0, cib-update=70, confirmed=true) ok
=====

WebSite stopped and won't start now.
=====
[root at an-c03n01 ~]# pcs status
Cluster name: an-cluster-03
Last updated: Wed Jan  1 18:23:44 2014
Last change: Wed Jan  1 18:22:32 2014 via crm_attribute on 
an-c03n01.alteeve.ca
Stack: corosync
Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
7 Resources configured


Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]

Full list of resources:

  fence_n01_virsh	(stonith:fence_virsh):	Started an-c03n02.alteeve.ca
  fence_n02_virsh	(stonith:fence_virsh):	Started an-c03n02.alteeve.ca
  ClusterIP	(ocf::heartbeat:IPaddr2):	Started an-c03n01.alteeve.ca
  Master/Slave Set: WebDataClone [WebData]
      Masters: [ an-c03n02.alteeve.ca ]
      Slaves: [ an-c03n01.alteeve.ca ]
  WebFS	(ocf::heartbeat:Filesystem):	Started an-c03n02.alteeve.ca
  WebSite	(ocf::heartbeat:apache):	Stopped

PCSD Status:
an-c03n01.alteeve.ca:
   an-c03n01.alteeve.ca: Online
an-c03n02.alteeve.ca:
   an-c03n02.alteeve.ca: Online

Daemon Status:
   corosync: active/disabled
   pacemaker: active/disabled
   pcsd: active/enabled
=====

Here is what I see when I try to start it manually:

=====
[root at an-c03n01 ~]# pcs resource debug-start WebSite --full
Operation start for WebSite (ocf:heartbeat:apache) returned 0
  >  stderr: + 18:24:17: 46: . /usr/lib/ocf/lib/heartbeat/apache-conf.sh
  >  stderr: + 18:24:17: 47: . /usr/lib/ocf/lib/heartbeat/http-mon.sh
  >  stderr: ++ 18:24:17: 18: bind_address=127.0.0.1
  >  stderr: ++ 18:24:17: 19: curl_ipv6_opts=
  >  stderr: ++ 18:24:17: 20: ocf_is_true ''
  >  stderr: ++ 18:24:17: ocf_is_true:101: case "$1" in
  >  stderr: ++ 18:24:17: ocf_is_true:103: false
  >  stderr: ++ 18:24:17: 20: grep -qs ::
  >  stderr: ++ 18:24:17: 20: echo ''
  >  stderr: ++ 18:24:17: 24: WGETOPTS='-O- -q -L --no-proxy 
--bind-address=127.0.0.1'
  >  stderr: ++ 18:24:17: 25: CURLOPTS='-o - -Ss -L --interface lo '
  >  stderr: + 18:24:17: 48: HA_VARRUNDIR=/var/run/
  >  stderr: + 18:24:17: 56: IBMHTTPD=/opt/IBMHTTPServer/bin/httpd
  >  stderr: + 18:24:17: 57: HTTPDLIST='/sbin/httpd2 /usr/sbin/httpd2 
/usr/sbin/apache2 /sbin/httpd /usr/sbin/httpd /usr/sbin/apache 
/opt/IBMHTTPServer/bin/httpd'
  >  stderr: + 18:24:17: 58: MPM=/usr/share/apache2/find_mpm
  >  stderr: + 18:24:17: 59: '[' -x /usr/share/apache2/find_mpm ']'
  >  stderr: + 18:24:17: 63: LOCALHOST=http://localhost
  >  stderr: + 18:24:17: 64: HTTPDOPTS=-DSTATUS
  >  stderr: + 18:24:17: 65: 
DEFAULT_IBMCONFIG=/opt/IBMHTTPServer/conf/httpd.conf
  >  stderr: + 18:24:17: 66: DEFAULT_SUSECONFIG=/etc/apache2/httpd.conf
  >  stderr: + 18:24:17: 67: DEFAULT_RHELCONFIG=/etc/httpd/conf/httpd.conf
  >  stderr: ++ 18:24:17: 79: basename 
/usr/lib/ocf/resource.d/heartbeat/apache
  >  stderr: + 18:24:17: 79: CMD=apache
  >  stderr: + 18:24:17: 654: OCF_REQUIRED_PARAMS=
  >  stderr: + 18:24:17: 655: OCF_REQUIRED_BINARIES=
  >  stderr: + 18:24:17: 656: ocf_rarun start
  >  stderr: + 18:24:17: ocf_rarun:137: mk_action_func
  >  stderr: ++ 18:24:17: mk_action_func:50: echo apache_start
  >  stderr: ++ 18:24:17: mk_action_func:50: tr - _
  >  stderr: + 18:24:17: mk_action_func:50: ACTION_FUNC=apache_start
  >  stderr: + 18:24:17: ocf_rarun:138: validate_args
  >  stderr: + 18:24:17: validate_args:53: is_function apache_start
  >  stderr: ++ 18:24:17: is_function:41: command -v apache_start
  >  stderr: + 18:24:17: is_function:41: test zapache_start = zapache_start
  >  stderr: + 18:24:17: ocf_rarun:139: simple_actions
  >  stderr: + 18:24:17: simple_actions:60: case $__OCF_ACTION in
  >  stderr: + 18:24:17: ocf_rarun:140: check_required_params
  >  stderr: + 18:24:17: check_required_params:74: local v
  >  stderr: + 18:24:17: ocf_rarun:141: run_function apache_getconfig
  >  stderr: + 18:24:17: run_function:44: is_function apache_getconfig
  >  stderr: ++ 18:24:17: is_function:41: command -v apache_getconfig
  >  stderr: + 18:24:17: is_function:41: test zapache_getconfig = 
zapache_getconfig
  >  stderr: + 18:24:17: run_function:44: apache_getconfig
  >  stderr: + 18:24:17: apache_getconfig:628: HTTPD=
  >  stderr: + 18:24:17: apache_getconfig:629: PORT=
  >  stderr: + 18:24:17: apache_getconfig:630: 
STATUSURL=http://127.0.0.1/server-status
  >  stderr: + 18:24:17: apache_getconfig:631: 
CONFIGFILE=/etc/httpd/conf/httpd.conf
  >  stderr: + 18:24:17: apache_getconfig:632: OPTIONS=
  >  stderr: + 18:24:17: apache_getconfig:633: CLIENT=
  >  stderr: + 18:24:17: apache_getconfig:634: TESTREGEX='</ *html *>'
  >  stderr: + 18:24:17: apache_getconfig:635: TESTURL=
  >  stderr: + 18:24:17: apache_getconfig:636: TESTREGEX10=
  >  stderr: + 18:24:17: apache_getconfig:637: TESTCONFFILE=
  >  stderr: + 18:24:17: apache_getconfig:638: TESTNAME=
  >  stderr: + 18:24:17: apache_getconfig:639: : /etc/apache2/envvars
  >  stderr: + 18:24:17: apache_getconfig:640: source_envfiles 
/etc/apache2/envvars
  >  stderr: + 18:24:17: source_envfiles:16: for f in '"$@"'
  >  stderr: + 18:24:17: source_envfiles:17: '[' -f /etc/apache2/envvars 
-a -r /etc/apache2/envvars ']'
  >  stderr: + 18:24:17: apache_getconfig:642: '[' X = X -o '!' -f '' -o 
'!' -x '' ']'
  >  stderr: + 18:24:17: apache_getconfig:643: find_httpd_prog
  >  stderr: + 18:24:17: find_httpd_prog:603: case $0 in
  >  stderr: + 18:24:17: find_httpd_prog:608: HTTPD=
  >  stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
  >  stderr: + 18:24:17: find_httpd_prog:611: '[' -f /sbin/httpd2 -a -x 
/sbin/httpd2 ']'
  >  stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
  >  stderr: + 18:24:17: find_httpd_prog:611: '[' -f /usr/sbin/httpd2 -a 
-x /usr/sbin/httpd2 ']'
  >  stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
  >  stderr: + 18:24:17: find_httpd_prog:611: '[' -f /usr/sbin/apache2 
-a -x /usr/sbin/apache2 ']'
  >  stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
  >  stderr: + 18:24:17: find_httpd_prog:611: '[' -f /sbin/httpd -a -x 
/sbin/httpd ']'
  >  stderr: + 18:24:17: find_httpd_prog:612: HTTPD=/sbin/httpd
  >  stderr: + 18:24:17: find_httpd_prog:613: break
  >  stderr: + 18:24:17: find_httpd_prog:618: '[' X '!=' X -a 
X/sbin/httpd '!=' X ']'
  >  stderr: ++ 18:24:17: find_httpd_prog:621: detect_default_config
  >  stderr: ++ 18:24:17: detect_default_config:387: '[' -f 
/etc/apache2/httpd.conf ']'
  >  stderr: ++ 18:24:17: detect_default_config:390: echo 
/etc/httpd/conf/httpd.conf
  >  stderr: + 18:24:17: find_httpd_prog:621: 
DefaultConfig=/etc/httpd/conf/httpd.conf
  >  stderr: + 18:24:17: apache_getconfig:646: 
CONFIGFILE=/etc/httpd/conf/httpd.conf
  >  stderr: ++ 18:24:17: apache_getconfig:647: basename /sbin/httpd
  >  stderr: + 18:24:17: apache_getconfig:647: httpd_basename=httpd
  >  stderr: + 18:24:17: apache_getconfig:648: case $httpd_basename in
  >  stderr: + 18:24:17: apache_getconfig:651: GetParams 
/etc/httpd/conf/httpd.conf
  >  stderr: + 18:24:17: GetParams:134: 
ConfigFile=/etc/httpd/conf/httpd.conf
  >  stderr: + 18:24:17: GetParams:135: '[' '!' -f 
/etc/httpd/conf/httpd.conf ']'
  >  stderr: + 18:24:17: GetParams:138: get_apache_params 
/etc/httpd/conf/httpd.conf ServerRoot PidFile Port Listen
  >  stderr: + 18:24:17: get_apache_params:79: 
configfile=/etc/httpd/conf/httpd.conf
  >  stderr: + 18:24:17: get_apache_params:80: shift 1
  >  stderr: ++ 18:24:17: get_apache_params:81: echo ServerRoot PidFile 
Port Listen
  >  stderr: ++ 18:24:17: get_apache_params:81: sed 's/ /,/g'
  >  stderr: + 18:24:17: get_apache_params:81: 
vars=ServerRoot,PidFile,Port,Listen
  >  stderr: ++ 18:24:17: get_apache_params:99: apachecat 
/etc/httpd/conf/httpd.conf
  >  stderr: ++ 18:24:17: get_apache_params:99: awk -v 
vars=ServerRoot,PidFile,Port,Listen '
  >  stderr: 	BEGIN{
  >  stderr: 		split(vars,v,",");
  >  stderr: 		for( i in v )
  >  stderr: 			vl[i]=tolower(v[i]);
  >  stderr: 	}
  >  stderr: 	{
  >  stderr: 		for( i in v )
  >  stderr: 			if( tolower($1)==vl[i] ) {
  >  stderr: 			print v[i]"="$2
  >  stderr: 			delete vl[i]
  >  stderr: 			break
  >  stderr: 		}
  >  stderr: 	}
  >  stderr: 	'
  >  stderr: ++ 18:24:17: apachecat:71: sed 
's/#.*//;s/[[:blank:]]*$//;s/^[[:blank:]]*//'
  >  stderr: ++ 18:24:17: apachecat:70: awk '
  >  stderr: 	function procline() {
  >  stderr: 		split($0,a);
  >  stderr: 		if( a[1]~/^[Ii]nclude$/ ) {
  >  stderr: 			procinclude(a[2]);
  >  stderr: 		} else {
  >  stderr: 			if( a[1]=="ServerRoot" ) {
  >  stderr: 				rootdir=a[2];
  >  stderr: 				gsub("\"","",rootdir);
  >  stderr: 			}
  >  stderr: 			print;
  >  stderr: 		}
  >  stderr: 	}
  >  stderr: 	function printfile(infile, a) {
  >  stderr: 		while( (getline<infile) > 0 ) {
  >  stderr: 			procline();
  >  stderr: 		}
  >  stderr: 		close(infile);
  >  stderr: 	}
  >  stderr: 	function allfiles(dir, cmd,f) {
  >  stderr: 		cmd="find -L "dir" -type f";
  >  stderr: 		while( ( cmd | getline f ) > 0 ) {
  >  stderr: 			printfile(f);
  >  stderr: 		}
  >  stderr: 		close(cmd);
  >  stderr: 	}
  >  stderr: 	function listfiles(pattern, cmd,f) {
  >  stderr: 		cmd="ls "pattern" 2>/dev/null";
  >  stderr: 		while( ( cmd | getline f ) > 0 ) {
  >  stderr: 			printfile(f);
  >  stderr: 		}
  >  stderr: 		close(cmd);
  >  stderr: 	}
  >  stderr: 	function procinclude(spec) {
  >  stderr: 		if( rootdir!="" && spec!~/^\// ) {
  >  stderr: 			spec=rootdir"/"spec;
  >  stderr: 		}
  >  stderr: 		if( isdir(spec) ) {
  >  stderr: 			allfiles(spec); # read all files in a directory (and 
subdirs)
  >  stderr: 		} else {
  >  stderr: 			listfiles(spec); # there could be jokers
  >  stderr: 		}
  >  stderr: 	}
  >  stderr: 	function isdir(s) {
  >  stderr: 		return !system("test -d \""s"\"");
  >  stderr: 	}
  >  stderr: 	{ procline(); }
  >  stderr: 	' /etc/httpd/conf/httpd.conf
  >  stderr: ++ 18:24:17: apachecat:72: grep -v '^$'
  >  stderr: + 18:24:17: get_apache_params:98: eval 
'ServerRoot="/etc/httpd"' Listen=80
  >  stderr: ++ 18:24:17: get_apache_params:98: ServerRoot=/etc/httpd
  >  stderr: ++ 18:24:17: get_apache_params:98: Listen=80
  >  stderr: + 18:24:17: GetParams:139: case $PidFile in
  >  stderr: + 18:24:17: GetParams:145: PidFile=/var/run//httpd.pid
  >  stderr: + 18:24:17: GetParams:149: PIDFILE_DIRECTIVE=true
  >  stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
  >  stderr: + 18:24:17: GetParams:154: CheckPort ''
  >  stderr: + 18:24:17: CheckPort:116: ocf_is_decimal ''
  >  stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
  >  stderr: + 18:24:17: ocf_is_decimal:94: false
  >  stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
  >  stderr: + 18:24:17: GetParams:154: CheckPort ''
  >  stderr: + 18:24:17: CheckPort:116: ocf_is_decimal ''
  >  stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
  >  stderr: + 18:24:17: ocf_is_decimal:94: false
  >  stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
  >  stderr: + 18:24:17: GetParams:154: CheckPort 80
  >  stderr: + 18:24:17: CheckPort:116: ocf_is_decimal 80
  >  stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
  >  stderr: + 18:24:17: ocf_is_decimal:96: true
  >  stderr: + 18:24:17: CheckPort:116: '[' 80 -gt 0 ']'
  >  stderr: + 18:24:17: GetParams:155: PORT=80
  >  stderr: + 18:24:17: GetParams:156: break
  >  stderr: + 18:24:17: GetParams:160: echo 80
  >  stderr: + 18:24:17: GetParams:160: grep :
  >  stderr: + 18:24:17: GetParams:161: Listen=localhost:80
  >  stderr: + 18:24:17: GetParams:183: '[' 
Xhttp://127.0.0.1/server-status = X ']'
  >  stderr: + 18:24:17: GetParams:189: test /var/run//httpd.pid
  >  stderr: + 18:24:17: GetParams:192: return 0
  >  stderr: + 18:24:17: ocf_rarun:142: validate_env
  >  stderr: + 18:24:17: validate_env:122: check_required_binaries
  >  stderr: + 18:24:17: check_required_binaries:113: local v
  >  stderr: + 18:24:17: validate_env:123: is_function apache_validate_all
  >  stderr: ++ 18:24:17: is_function:41: command -v apache_validate_all
  >  stderr: + 18:24:17: is_function:41: test zapache_validate_all = 
zapache_validate_all
  >  stderr: + 18:24:17: validate_env:125: local rc
  >  stderr: + 18:24:17: validate_env:126: LSB_STATUS_STOPPED=3
  >  stderr: + 18:24:17: validate_env:127: apache_validate_all
  >  stderr: + 18:24:17: apache_validate_all:573: CheckPort 80
  >  stderr: + 18:24:17: CheckPort:116: ocf_is_decimal 80
  >  stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
  >  stderr: + 18:24:17: ocf_is_decimal:96: true
  >  stderr: + 18:24:17: CheckPort:116: '[' 80 -gt 0 ']'
  >  stderr: + 18:24:17: apache_validate_all:575: : OK
  >  stderr: + 18:24:17: apache_validate_all:581: case $STATUSURL in
  >  stderr: + 18:24:17: apache_validate_all:588: '[' '!' -x /sbin/httpd ']'
  >  stderr: + 18:24:17: apache_validate_all:593: '[' '!' -f 
/etc/httpd/conf/httpd.conf ']'
  >  stderr: + 18:24:17: apache_validate_all:599: return 0
  >  stderr: + 18:24:17: validate_env:128: rc=0
  >  stderr: + 18:24:17: validate_env:129: '[' 0 -ne 0 ']'
  >  stderr: + 18:24:17: ocf_rarun:143: ocf_is_probe
  >  stderr: + 18:24:17: ocf_is_probe:438: '[' start = monitor -a 0 = 0 ']'
  >  stderr: + 18:24:17: ocf_rarun:144: shift 1
  >  stderr: + 18:24:17: ocf_rarun:145: apache_start
  >  stderr: + 18:24:17: apache_start:164: silent_status
  >  stderr: + 18:24:17: silent_status:125: '[' -f /var/run//httpd.pid ']'
  >  stderr: + 18:24:17: silent_status:128: : No pid file
  >  stderr: + 18:24:17: silent_status:129: false
  >  stderr: + 18:24:17: apache_start:170: validate_default_config
  >  stderr: + 18:24:17: validate_default_config:135: '[' -e 
/etc/SuSE-release ']'
  >  stderr: + 18:24:17: validate_default_config:138: return 0
  >  stderr: + 18:24:17: apache_start:172: '[' -d /var/run/apache2 ']'
  >  stderr: + 18:24:17: apache_start:174: '[' -z true ']'
  >  stderr: + 18:24:17: apache_start:177: ocf_run /sbin/httpd -DSTATUS 
-f /etc/httpd/conf/httpd.conf -c 'PidFile /var/run//httpd.pid'
  >  stderr: + 18:24:17: ocf_run:361: local rc
  >  stderr: + 18:24:17: ocf_run:362: local output
  >  stderr: + 18:24:17: ocf_run:363: local verbose=1
  >  stderr: + 18:24:17: ocf_run:364: local loglevel=err
  >  stderr: + 18:24:17: ocf_run:365: local var
  >  stderr: + 18:24:17: ocf_run:367: for var in 1 2
  >  stderr: + 18:24:17: ocf_run:369: case "$1" in
  >  stderr: + 18:24:17: ocf_run:367: for var in 1 2
  >  stderr: + 18:24:17: ocf_run:369: case "$1" in
  >  stderr: ++ 18:24:17: ocf_run:381: /sbin/httpd -DSTATUS -f 
/etc/httpd/conf/httpd.conf -c 'PidFile /var/run//httpd.pid'
  >  stderr: + 18:24:17: ocf_run:381: output=
  >  stderr: + 18:24:17: ocf_run:382: rc=0
  >  stderr: ++ 18:24:17: ocf_run:383: echo
  >  stderr: + 18:24:17: ocf_run:383: output=
  >  stderr: + 18:24:17: ocf_run:384: '[' 0 -eq 0 ']'
  >  stderr: + 18:24:17: ocf_run:385: '[' 1 -a '!' -z '' ']'
  >  stderr: + 18:24:17: ocf_run:388: return 0
  >  stderr: + 18:24:17: apache_start:180: tries=0
  >  stderr: + 18:24:17: apache_start:181: :
  >  stderr: + 18:24:17: apache_start:183: apache_monitor
  >  stderr: + 18:24:17: apache_monitor:367: silent_status
  >  stderr: + 18:24:17: silent_status:125: '[' -f /var/run//httpd.pid ']'
  >  stderr: ++ 18:24:17: silent_status:126: cat /var/run//httpd.pid
  >  stderr: + 18:24:17: silent_status:126: ProcessRunning 16404
  >  stderr: + 18:24:17: ProcessRunning:114: ApachePID=16404
  >  stderr: + 18:24:17: ProcessRunning:116: '[' -d /proc -a -d /proc/1 ']'
  >  stderr: + 18:24:17: ProcessRunning:117: '[' -d /proc/16404 ']'
  >  stderr: + 18:24:17: apache_monitor:368: '[' 0 -ne 0 ']'
  >  stderr: ++ 18:24:17: apache_monitor:373: findhttpclient
  >  stderr: ++ 18:24:17: findhttpclient:66: '[' x '!=' x ']'
  >  stderr: ++ 18:24:17: findhttpclient:68: which wget
  >  stderr: ++ 18:24:17: findhttpclient:69: echo wget
  >  stderr: + 18:24:17: apache_monitor:373: ourhttpclient=wget
  >  stderr: + 18:24:17: apache_monitor:374: '[' -z wget ']'
  >  stderr: + 18:24:17: apache_monitor:379: case `ocf_check_level 10` in
  >  stderr: ++ 18:24:17: apache_monitor:379: ocf_check_level 10
  >  stderr: ++ 18:24:17: ocf_check_level:643: local lvl prev
  >  stderr: ++ 18:24:17: ocf_check_level:644: lvl=0
  >  stderr: ++ 18:24:17: ocf_check_level:645: prev=0
  >  stderr: ++ 18:24:17: ocf_check_level:646: ocf_is_decimal 0
  >  stderr: ++ 18:24:17: ocf_is_decimal:92: case "$1" in
  >  stderr: ++ 18:24:17: ocf_is_decimal:96: true
  >  stderr: ++ 18:24:17: ocf_check_level:648: for lvl in '"$@"'
  >  stderr: ++ 18:24:17: ocf_check_level:649: '[' 10 -eq 0 ']'
  >  stderr: ++ 18:24:17: ocf_check_level:651: '[' 10 -gt 0 ']'
  >  stderr: ++ 18:24:17: ocf_check_level:652: lvl=0
  >  stderr: ++ 18:24:17: ocf_check_level:653: break
  >  stderr: ++ 18:24:17: ocf_check_level:658: echo 0
  >  stderr: + 18:24:17: apache_monitor:380: apache_monitor_basic
  >  stderr: + 18:24:17: apache_monitor_basic:351: wget_func 
http://127.0.0.1/server-status
  >  stderr: + 18:24:17: apache_monitor_basic:351: grep -Ei '</ *html *>'
  >  stderr: + 18:24:17: wget_func:49: auth=
  >  stderr: + 18:24:17: wget_func:50: cl_opts='-O- -q -L --no-proxy 
--bind-address=127.0.0.1 '
  >  stderr: + 18:24:17: wget_func:51: '[' x '!=' x ']'
  >  stderr: + 18:24:17: wget_func:53: wget -O- -q -L --no-proxy 
--bind-address=127.0.0.1 http://127.0.0.1/server-status
  >  stderr: + 18:24:17: apache_monitor_basic:353: return 0
  >  stderr: + 18:24:17: apache_start:184: ec=0
  >  stderr: + 18:24:17: apache_start:185: '[' 0 -eq 7 ']'
  >  stderr: + 18:24:17: apache_start:191: break
  >  stderr: + 18:24:17: apache_start:195: '[' 0 -ne 0 ']'
  >  stderr: + 18:24:17: apache_start:198: return 0
=====

Only log entry is on an-c03n01 (where I ran the command):

=====
Jan  1 18:24:17 an-c03n01 crm_resource[16170]: notice: crm_log_args: 
Invoked: /usr/sbin/crm_resource -r WebSite --force-start -V
=====

When I delete the constraint that was added when the node went into 
standby, WebSite starts.

=====
[root at an-c03n01 ~]# pcs config show | grep uname -B 1
       Rule: score=-INFINITY role=Master 
(id:drbd-fence-by-handler-r0-rule-WebDataClone)
         Expression: #uname ne an-c03n02.alteeve.ca 
(id:drbd-fence-by-handler-r0-expr-WebDataClone)
[root at an-c03n01 ~]# pcs constraint delete 
drbd-fence-by-handler-r0-rule-WebDataClone
=====

==[an-c03n01 logs]===
Jan  1 18:26:44 an-c03n01 cibadmin[17037]: notice: crm_log_args: 
Invoked: /usr/sbin/cibadmin -c -R --xml-text <constraints>
   <rsc_location id="cli-prefer-WebDataClone" 
node="an-c03n01.alteeve.ca" role="Started" rsc="WebDataClone" 
score="INFINITY"/>
   <rsc_location id="cli-prefer-ClusterIP" node="an-c03n01.alteeve.ca" 
role="Started" rsc="ClusterIP" score="INFINITY"/>
   <rsc_colocation id="colocation-WebFS-WebDataClone-INFINITY" 
rsc="WebFS" score="INFINITY" with-rsc="WebDataClone" 
with-rsc-role="Master"/>
   <rsc_order first="WebDataClone" first-ac
Jan  1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.192.9
Jan  1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.193.1 
695289e37618e5e9a184d2bc8c90d421
Jan  1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- 
<rsc_location rsc="WebDataClone" id="drbd-fence-by-handler-r0-WebDataClone">
Jan  1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: --         <rule 
role="Master" score="-INFINITY" 
id="drbd-fence-by-handler-r0-rule-WebDataClone">
Jan  1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- 
<expression attribute="#uname" operation="ne" 
value="an-c03n02.alteeve.ca" 
id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
Jan  1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: --         </rule>
Jan  1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- 
</rsc_location>
Jan  1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: ++ <cib 
admin_epoch="0" cib-last-written="Wed Jan  1 18:26:44 2014" 
crm_feature_set="3.0.7" epoch="193" have-quorum="1" num_updates="1" 
update-client="cibadmin" update-origin="an-c03n01.alteeve.ca" 
validate-with="pacemaker-1.2" dc-uuid="1"/>
Jan  1 18:26:44 an-c03n01 crmd[1767]: notice: do_state_transition: State 
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC 
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jan  1 18:26:44 an-c03n01 stonith-ng[1763]: notice: unpack_config: On 
loss of CCM Quorum: Ignore
Jan  1 18:26:44 an-c03n01 crmd[1767]: notice: do_state_transition: State 
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC 
cause=C_FSA_INTERNAL origin=do_election_check ]
Jan  1 18:26:44 an-c03n01 stonith-ng[1763]: notice: 
stonith_device_register: Added 'fence_n01_virsh' to the device list (2 
active devices)
Jan  1 18:26:44 an-c03n01 stonith-ng[1763]: notice: 
stonith_device_register: Added 'fence_n02_virsh' to the device list (2 
active devices)
Jan  1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_local_callback: 
Sending full refresh (origin=crmd)
Jan  1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: master-WebData (10000)
Jan  1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: probe_complete (true)
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Promote 
WebData:0	(Slave -> Master an-c03n01.alteeve.ca)
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Demote 
WebData:1	(Master -> Slave an-c03n02.alteeve.ca)
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Move 
WebFS	(Started an-c03n02.alteeve.ca -> an-c03n01.alteeve.ca)
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebSite	(an-c03n01.alteeve.ca)
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 45: /var/lib/pacemaker/pengine/pe-input-183.bz2
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 2: cancel WebData_cancel_60000 on an-c03n01.alteeve.ca (local)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 47: stop WebFS_stop_0 on an-c03n02.alteeve.ca
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 67: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca 
(local)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 69: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=111, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: run_graph: Transition 45 
(Complete=5, Pending=0, Fired=0, Skipped=15, Incomplete=10, 
Source=/var/lib/pacemaker/pengine/pe-input-183.bz2): Stopped
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: unpack_config: On loss 
of CCM Quorum: Ignore
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Promote 
WebData:0	(Slave -> Master an-c03n01.alteeve.ca)
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Demote 
WebData:1	(Master -> Slave an-c03n02.alteeve.ca)
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebFS	(an-c03n01.alteeve.ca)
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start 
WebSite	(an-c03n01.alteeve.ca)
Jan  1 18:26:46 an-c03n01 pengine[1766]: notice: process_pe_message: 
Calculated Transition 46: /var/lib/pacemaker/pengine/pe-input-184.bz2
Jan  1 18:26:46 an-c03n01 crmd[1767]: warning: destroy_action: 
Cancelling timer for action 2 (src=476)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 64: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca 
(local)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 66: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=112, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 18: demote WebData_demote_0 on an-c03n02.alteeve.ca
Jan  1 18:26:46 an-c03n01 kernel: [ 3524.517000] block drbd0: peer( 
Primary -> Secondary )
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 65: notify WebData_post_notify_demote_0 on an-c03n01.alteeve.ca 
(local)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 67: notify WebData_post_notify_demote_0 on an-c03n02.alteeve.ca
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=113, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 60: notify WebData_pre_notify_promote_0 on an-c03n01.alteeve.ca 
(local)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 62: notify WebData_pre_notify_promote_0 on an-c03n02.alteeve.ca
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=114, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 16: promote WebData_promote_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:26:46 an-c03n01 kernel: [ 3524.605607] block drbd0: role( 
Secondary -> Primary )
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_promote_0 (call=115, rc=0, cib-update=324, 
confirmed=true) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 61: notify WebData_post_notify_promote_0 on an-c03n01.alteeve.ca 
(local)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 63: notify WebData_post_notify_promote_0 on an-c03n02.alteeve.ca
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=116, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 45: start WebFS_start_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 20: monitor WebData_monitor_60000 on an-c03n02.alteeve.ca
Jan  1 18:26:46 an-c03n01 Filesystem(WebFS)[17211]: INFO: Running start 
for /dev/drbd0 on /var/www/html
Jan  1 18:26:46 an-c03n01 kernel: [ 3524.688633] EXT4-fs (drbd0): 
mounted filesystem with ordered data mode. Opts: (null)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebFS_start_0 (call=117, rc=0, cib-update=325, confirmed=true) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 46: monitor WebFS_monitor_60000 on an-c03n01.alteeve.ca (local)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 47: start WebSite_start_0 on an-c03n01.alteeve.ca (local)
Jan  1 18:26:46 an-c03n01 apache(WebSite)[17276]: INFO: apache already 
running (pid 16404)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebSite_start_0 (call=119, rc=0, cib-update=326, 
confirmed=true) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebFS_monitor_60000 (call=118, rc=0, cib-update=327, 
confirmed=false) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating 
action 48: monitor WebSite_monitor_60000 on an-c03n01.alteeve.ca (local)
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM 
operation WebSite_monitor_60000 (call=120, rc=0, cib-update=328, 
confirmed=false) ok
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: run_graph: Transition 46 
(Complete=27, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-184.bz2): Complete
Jan  1 18:26:46 an-c03n01 crmd[1767]: notice: do_state_transition: State 
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS 
cause=C_FSA_INTERNAL origin=notify_crmd ]

==[an-c03n02 logs]===
Jan  1 18:26:44 an-c03n02 crmd[16388]: notice: do_state_transition: 
State transition S_NOT_DC -> S_PENDING [ input=I_PENDING 
cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Jan  1 18:26:44 an-c03n02 stonith-ng[16384]: notice: unpack_config: On 
loss of CCM Quorum: Ignore
Jan  1 18:26:44 an-c03n02 stonith-ng[16384]: notice: 
stonith_device_register: Device 'fence_n01_virsh' already existed in 
device list (2 active devices)
Jan  1 18:26:44 an-c03n02 stonith-ng[16384]: notice: 
stonith_device_register: Device 'fence_n02_virsh' already existed in 
device list (2 active devices)
Jan  1 18:26:45 an-c03n02 crmd[16388]: notice: do_state_transition: 
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC 
cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jan  1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_local_callback: 
Sending full refresh (origin=crmd)
Jan  1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: master-WebData (10000)
Jan  1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: probe_complete (true)
Jan  1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=112, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: Running stop 
for /dev/drbd0 on /var/www/html
Jan  1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: Trying to 
unmount /var/www/html
Jan  1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: unmounted 
/var/www/html successfully
Jan  1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebFS_stop_0 (call=111, rc=0, cib-update=73, confirmed=true) ok
Jan  1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=113, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n02 kernel: [20086.520840] block drbd0: role( 
Primary -> Secondary )
Jan  1 18:26:46 an-c03n02 kernel: [20086.520911] block drbd0: bitmap 
WRITE of 0 pages took 0 jiffies
Jan  1 18:26:46 an-c03n02 kernel: [20086.520914] block drbd0: 0 KB (0 
bits) marked out-of-sync by on disk bit-map.
Jan  1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_demote_0 (call=114, rc=0, cib-update=74, 
confirmed=true) ok
Jan  1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=115, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=116, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n02 kernel: [20086.610152] block drbd0: peer( 
Secondary -> Primary )
Jan  1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_notify_0 (call=117, rc=0, cib-update=0, confirmed=true) ok
Jan  1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM 
operation WebData_monitor_60000 (call=118, rc=0, cib-update=75, 
confirmed=false) ok
=====

Now WebSite is working:
=====
[root at an-c03n01 ~]# pcs status
Cluster name: an-cluster-03
Last updated: Wed Jan  1 18:28:04 2014
Last change: Wed Jan  1 18:26:44 2014 via cibadmin on an-c03n01.alteeve.ca
Stack: corosync
Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
7 Resources configured


Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]

Full list of resources:

  fence_n01_virsh	(stonith:fence_virsh):	Started an-c03n02.alteeve.ca
  fence_n02_virsh	(stonith:fence_virsh):	Started an-c03n02.alteeve.ca
  ClusterIP	(ocf::heartbeat:IPaddr2):	Started an-c03n01.alteeve.ca
  Master/Slave Set: WebDataClone [WebData]
      Masters: [ an-c03n01.alteeve.ca ]
      Slaves: [ an-c03n02.alteeve.ca ]
  WebFS	(ocf::heartbeat:Filesystem):	Started an-c03n01.alteeve.ca
  WebSite	(ocf::heartbeat:apache):	Started an-c03n01.alteeve.ca

PCSD Status:
an-c03n01.alteeve.ca:
   an-c03n01.alteeve.ca: Online
an-c03n02.alteeve.ca:
   an-c03n02.alteeve.ca: Online

Daemon Status:
   corosync: active/disabled
   pacemaker: active/disabled
   pcsd: active/enabled
=====

Is this a bug?

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without 
access to education?




More information about the Pacemaker mailing list