[Pacemaker] pacemaker 1.1.10 + RHEL 7 beta issues
Chris Feist
cfeist at redhat.com
Thu Jan 2 17:42:24 UTC 2014
On 01/01/2014 05:30 PM, Digimer wrote:
> I have a problem where putting a node into standby causes my "WebSite" resource
> to stop and refuse to start. This is the reproducer I show here, but I have seen
> this happen with DRBD earlier (before I realized what was wrong, so I didn't
> document it), so I don't think this is an apache RA issue.
>
...snip...
I think this is where the issue is. Location constraints requiring the
resources to be on an-c03n01 have been created, so when that node is put on
standby, they can't run anywhere else.
It looks like these may have been created by a 'pcs resource move' command. Can
you try, 'pcs resource clear <resource name>' for each resource.
Thanks!
Chris
> Location Constraints:
> Resource: ClusterIP
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-ClusterIP)
> Resource: WebDataClone
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-WebDataClone)
> Resource: WebSite
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-WebSite)
> Put an-c03n01 into standby:
>
> =====
> pcs cluster standby an-c03n01.alteeve.ca
>
> ==[an-c03n01 logs]===
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.189.16
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.190.1
> 87f59f171d277d9cb1b51cbda41fbf24
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: -- <cib admin_epoch="0"
> epoch="189" num_updates="16"/>
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++ <instance_attributes
> id="nodes-1">
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++ <nvpair
> id="nodes-1-standby" name="standby" value="on"/>
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++ </instance_attributes>
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move ClusterIP
> (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Demote WebData:0
> (Master -> Stopped an-c03n01.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:1 (Slave -> Master an-c03n02.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move WebFS
> (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move WebSite
> (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 37: /var/lib/pacemaker/pengine/pe-input-176.bz2
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 15: stop ClusterIP_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 6: cancel WebData_cancel_60000 on an-c03n02.alteeve.ca
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 50: stop WebSite_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 64: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 66: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:31 an-c03n01 apache(WebSite)[14743]: INFO: Attempting graceful stop
> of apache PID 10641
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=97, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:31 an-c03n01 IPaddr2(ClusterIP)[14742]: INFO: IP status = ok, IP_CIP=
> Jan 1 18:16:31 an-c03n01 avahi-daemon[298]: Withdrawing address record for
> 192.168.122.10 on eth0.
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> ClusterIP_stop_0 (call=94, rc=0, cib-update=283, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 apache(WebSite)[14743]: INFO: apache stopped.
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebSite_stop_0 (call=96, rc=0, cib-update=284, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: run_graph: Transition 37
> (Complete=6, Pending=0, Fired=0, Skipped=22, Incomplete=13,
> Source=/var/lib/pacemaker/pengine/pe-input-176.bz2): Stopped
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start ClusterIP
> (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Demote WebData:0
> (Master -> Stopped an-c03n01.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:1 (Slave -> Master an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Move WebFS
> (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start WebSite
> (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 38: /var/lib/pacemaker/pengine/pe-input-177.bz2
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 12: start ClusterIP_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 43: stop WebFS_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 59: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 61: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=100, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: Running stop for
> /dev/drbd0 on /var/www/html
> Jan 1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: Trying to unmount
> /var/www/html
> Jan 1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: unmounted
> /var/www/html successfully
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebFS_stop_0 (call=99, rc=0, cib-update=286, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 14: demote WebData_demote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 13: monitor ClusterIP_monitor_30000 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.547981] block drbd0: role( Primary ->
> Secondary )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.548034] block drbd0: bitmap WRITE of 0
> pages took 0 jiffies
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.548037] block drbd0: 0 KB (0 bits)
> marked out-of-sync by on disk bit-map.
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_demote_0 (call=101, rc=0, cib-update=287, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 60: notify WebData_post_notify_demote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 62: notify WebData_post_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=102, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 54: notify WebData_pre_notify_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 55: notify WebData_pre_notify_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=103, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 15: stop WebData_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638480] drbd r0: peer( Secondary ->
> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638490] drbd r0: asender terminated
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638492] drbd r0: Terminating drbd_a_r0
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638732] drbd r0: Connection closed
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638741] drbd r0: conn( Disconnecting ->
> StandAlone )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638742] drbd r0: receiver terminated
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638743] drbd r0: Terminating drbd_r_r0
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638776] block drbd0: disk( UpToDate ->
> Failed )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651161] block drbd0: bitmap WRITE of 0
> pages took 0 jiffies
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651164] block drbd0: 0 KB (0 bits)
> marked out-of-sync by on disk bit-map.
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651167] block drbd0: disk( Failed ->
> Diskless )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651239] block drbd0: drbd_bm_resize
> called with capacity == 0
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651279] drbd r0: Terminating drbd_w_r0
> Jan 1 18:16:33 an-c03n01 attrd[1765]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-WebData (<null>)
> Jan 1 18:16:33 an-c03n01 attrd[1765]: notice: attrd_perform_update: Sent delete
> 154: node=1, attr=master-WebData, id=<n/a>, set=(null), section=status
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_stop_0 (call=104, rc=0, cib-update=288, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 56: notify WebData_post_notify_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: run_graph: Transition 38
> (Complete=24, Pending=0, Fired=0, Skipped=10, Incomplete=4,
> Source=/var/lib/pacemaker/pengine/pe-input-177.bz2): Stopped
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:0 (Slave -> Master an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start WebFS
> (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start WebSite
> (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 39: /var/lib/pacemaker/pengine/pe-input-178.bz2
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:0 (Slave -> Master an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start WebFS
> (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start WebSite
> (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 40: /var/lib/pacemaker/pengine/pe-input-178.bz2
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 54: notify WebData_pre_notify_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 16: promote WebData_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: handle_request: Current ping
> state: S_TRANSITION_ENGINE
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.190.11
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.191.1
> ba2e8658b93b742e3ef335fa93b50c26
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: -- <cib admin_epoch="0"
> epoch="190" num_updates="11"/>
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ <rsc_location
> rsc="WebDataClone" id="drbd-fence-by-handler-r0-WebDataClone">
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ <rule
> role="Master" score="-INFINITY" id="drbd-fence-by-handler-r0-rule-WebDataClone">
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ <expression
> attribute="#uname" operation="ne" value="an-c03n02.alteeve.ca"
> id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ </rule>
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ </rsc_location>
> Jan 1 18:16:34 an-c03n01 stonith-ng[1763]: notice: unpack_config: On loss of
> CCM Quorum: Ignore
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 55: notify WebData_post_notify_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 stonith-ng[1763]: notice: stonith_device_register:
> Added 'fence_n01_virsh' to the device list (2 active devices)
> Jan 1 18:16:34 an-c03n01 stonith-ng[1763]: notice: stonith_device_register:
> Added 'fence_n02_virsh' to the device list (2 active devices)
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 40
> (Complete=9, Pending=0, Fired=0, Skipped=4, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-178.bz2): Stopped
> Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: LogActions: Start WebFS
> (an-c03n02.alteeve.ca)
> Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: LogActions: Start WebSite
> (an-c03n02.alteeve.ca)
> Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 41: /var/lib/pacemaker/pengine/pe-input-179.bz2
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 42: start WebFS_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 43: monitor WebFS_monitor_60000 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 44: start WebSite_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 45: monitor WebSite_monitor_60000 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 41
> (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-179.bz2): Complete
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
>
> ==[an-c03n02 logs]===
> Jan 1 18:16:31 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=90, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=92, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO: Adding inet address
> 192.168.122.10/32 to device eth0
> Jan 1 18:16:33 an-c03n02 avahi-daemon[295]: Registering new address record for
> 192.168.122.10 on eth0.IPv4.
> Jan 1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO: Bringing device eth0 up
> Jan 1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO:
> /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
> /var/run/resource-agents/send_arp-192.168.122.10 eth0 192.168.122.10 auto
> not_used not_used
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> ClusterIP_start_0 (call=91, rc=0, cib-update=62, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 kernel: [19473.555292] block drbd0: peer( Primary ->
> Secondary )
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> ClusterIP_monitor_30000 (call=93, rc=0, cib-update=63, confirmed=false) ok
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=94, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=95, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645249] drbd r0: peer( Secondary ->
> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> DUnknown )
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645267] drbd r0: asender terminated
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645269] drbd r0: Terminating drbd_a_r0
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645531] drbd r0: Connection closed
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645538] drbd r0: conn( TearDown ->
> Unconnected )
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645539] drbd r0: receiver terminated
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645540] drbd r0: Restarting receiver
> thread
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645541] drbd r0: receiver (re)started
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645546] drbd r0: conn( Unconnected ->
> WFConnection )
> Jan 1 18:16:33 an-c03n02 attrd[16386]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-WebData (1000)
> Jan 1 18:16:33 an-c03n02 attrd[16386]: notice: attrd_perform_update: Sent
> update 152: master-WebData=1000
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=96, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=97, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 kernel: [19473.791812] drbd r0: helper command:
> /sbin/drbdadm fence-peer r0
> Jan 1 18:16:34 an-c03n02 crm-fence-peer.sh[32287]: invoked for r0
> Jan 1 18:16:34 an-c03n02 cibadmin[32319]: notice: crm_log_args: Invoked:
> cibadmin -C -o constraints -X <rsc_location rsc="WebDataClone"
> id="drbd-fence-by-handler-r0-WebDataClone">
> <rule role="Master" score="-INFINITY"
> id="drbd-fence-by-handler-r0-rule-WebDataClone">
> <expression attribute="#uname" operation="ne" value="an-c03n02.alteeve.ca"
> id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
> </rule>
> </rsc_location>
> Jan 1 18:16:34 an-c03n02 kernel: [19473.836661] drbd r0: helper command:
> /sbin/drbdadm fence-peer r0 exit code 4 (0x400)
> Jan 1 18:16:34 an-c03n02 kernel: [19473.836664] drbd r0: fence-peer helper
> returned 4 (peer was fenced)
> Jan 1 18:16:34 an-c03n02 kernel: [19473.836676] drbd r0: pdsk( DUnknown ->
> Outdated )
> Jan 1 18:16:34 an-c03n02 kernel: [19473.836681] block drbd0: role( Secondary ->
> Primary )
> Jan 1 18:16:34 an-c03n02 stonith-ng[16384]: notice: unpack_config: On loss of
> CCM Quorum: Ignore
> Jan 1 18:16:34 an-c03n02 crm-fence-peer.sh[32287]: INFO peer is reachable, my
> disk is UpToDate: placed constraint 'drbd-fence-by-handler-r0-WebDataClone'
> Jan 1 18:16:34 an-c03n02 kernel: [19473.837205] block drbd0: new current UUID
> 3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_promote_0 (call=98, rc=0, cib-update=64, confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 stonith-ng[16384]: notice: stonith_device_register:
> Device 'fence_n01_virsh' already existed in device list (2 active devices)
> Jan 1 18:16:34 an-c03n02 attrd[16386]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-WebData (10000)
> Jan 1 18:16:34 an-c03n02 attrd[16386]: notice: attrd_perform_update: Sent
> update 154: master-WebData=10000
> Jan 1 18:16:34 an-c03n02 stonith-ng[16384]: notice: stonith_device_register:
> Device 'fence_n02_virsh' already existed in device list (2 active devices)
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=99, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 Filesystem(WebFS)[32362]: INFO: Running start for
> /dev/drbd0 on /var/www/html
> Jan 1 18:16:34 an-c03n02 kernel: [19473.948256] EXT4-fs (drbd0): mounted
> filesystem with ordered data mode. Opts: (null)
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebFS_start_0 (call=100, rc=0, cib-update=65, confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebFS_monitor_60000 (call=101, rc=0, cib-update=66, confirmed=false) ok
> Jan 1 18:16:34 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:16:34 an-c03n02 apache(WebSite)[32427]: INFO: AH00558: httpd: Could
> not reliably determine the server's fully qualified domain name, using
> an-c03n02.alteeve.ca. Set the 'ServerName' directive globally to suppress this
> message
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebSite_start_0 (call=102, rc=0, cib-update=67, confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebSite_monitor_60000 (call=103, rc=0, cib-update=68, confirmed=false) ok
> Jan 1 18:16:43 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:16:53 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:03 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:13 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:23 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:33 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:43 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:53 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:18:03 an-c03n02 systemd: pacemaker.service: Got notification message
> from PID 32497, but reception only permitted for PID 16382
> =====
>
> Not sure why this happened:
> =====
> Jan 1 18:16:34 an-c03n02 cibadmin[32319]: notice: crm_log_args: Invoked:
> cibadmin -C -o constraints -X <rsc_location rsc="WebDataClone"
> id="drbd-fence-by-handler-r0-WebDataClone">
> <rule role="Master" score="-INFINITY"
> id="drbd-fence-by-handler-r0-rule-WebDataClone">
> <expression attribute="#uname" operation="ne" value="an-c03n02.alteeve.ca"
> id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
> </rule>
> </rsc_location>
> =====
>
> That seems to be the problem, but let me continue.
>
> Everything fails over properly:
>
> =====
> [root at an-c03n01 ~]# pcs status
> Cluster name: an-cluster-03
> Last updated: Wed Jan 1 18:19:01 2014
> Last change: Wed Jan 1 18:16:34 2014 via cibadmin on an-c03n02.alteeve.ca
> Stack: corosync
> Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
> Version: 1.1.10-19.el7-368c726
> 2 Nodes configured
> 7 Resources configured
>
>
> Node an-c03n01.alteeve.ca (1): standby
> Online: [ an-c03n02.alteeve.ca ]
>
> Full list of resources:
>
> fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n02.alteeve.ca
> Master/Slave Set: WebDataClone [WebData]
> Masters: [ an-c03n02.alteeve.ca ]
> Stopped: [ an-c03n01.alteeve.ca ]
> WebFS (ocf::heartbeat:Filesystem): Started an-c03n02.alteeve.ca
> WebSite (ocf::heartbeat:apache): Started an-c03n02.alteeve.ca
>
> PCSD Status:
> an-c03n01.alteeve.ca:
> an-c03n01.alteeve.ca: Online
> an-c03n02.alteeve.ca:
> an-c03n02.alteeve.ca: Online
>
> Daemon Status:
> corosync: active/disabled
> pacemaker: active/disabled
> pcsd: active/enabled
> =====
>
> Current config:
> =====
> [root at an-c03n01 ~]# pcs config show
> Cluster Name: an-cluster-03
> Corosync Nodes:
> an-c03n01.alteeve.ca an-c03n02.alteeve.ca
> Pacemaker Nodes:
> an-c03n01.alteeve.ca an-c03n02.alteeve.ca
>
> Resources:
> Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
> Attributes: ip=192.168.122.10 cidr_netmask=32
> Operations: monitor interval=30s (ClusterIP-monitor-interval-30s)
> Master: WebDataClone
> Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1
> notify=true
> Resource: WebData (class=ocf provider=linbit type=drbd)
> Attributes: drbd_resource=r0
> Operations: monitor interval=60s (WebData-monitor-60s)
> Resource: WebFS (class=ocf provider=heartbeat type=Filesystem)
> Attributes: device=/dev/drbd0 directory=/var/www/html fstype=ext4
> Operations: monitor interval=60s (WebFS-monitor-interval-60s)
> Resource: WebSite (class=ocf provider=heartbeat type=apache)
> Attributes: configfile=/etc/httpd/conf/httpd.conf
> statusurl=http://127.0.0.1/server-status
> Operations: monitor interval=60s (WebSite-monitor-interval-60s)
>
> Stonith Devices:
> Resource: fence_n01_virsh (class=stonith type=fence_virsh)
> Attributes: pcmk_host_list=an-c03n01.alteeve.ca ipaddr=lemass login=root
> passwd_script=/root/lemass.pw delay=15 port=an-c03n01
> Operations: monitor interval=60s (fence_n01_virsh-monitor-interval-60s)
> Resource: fence_n02_virsh (class=stonith type=fence_virsh)
> Attributes: pcmk_host_list=an-c03n02.alteeve.ca ipaddr=lemass login=root
> passwd_script=/root/lemass.pw port=an-c03n02
> Operations: monitor interval=60s (fence_n02_virsh-monitor-interval-60s)
> Fencing Levels:
>
> Location Constraints:
> Resource: ClusterIP
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-ClusterIP)
> Resource: WebDataClone
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-WebDataClone)
> Constraint: drbd-fence-by-handler-r0-WebDataClone
> Rule: score=-INFINITY role=Master
> (id:drbd-fence-by-handler-r0-rule-WebDataClone)
> Expression: #uname ne an-c03n02.alteeve.ca
> (id:drbd-fence-by-handler-r0-expr-WebDataClone)
> Resource: WebSite
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-WebSite)
> Ordering Constraints:
> promote WebDataClone then start WebFS (Mandatory)
> (id:order-WebDataClone-WebFS-mandatory)
> start WebFS then start WebSite (Mandatory) (id:order-WebFS-WebSite-mandatory)
> Colocation Constraints:
> WebFS with WebDataClone (INFINITY) (with-rsc-role:Master)
> (id:colocation-WebFS-WebDataClone-INFINITY)
> WebSite with ClusterIP (INFINITY) (id:colocation-WebSite-ClusterIP-INFINITY)
> WebSite with WebFS (INFINITY) (id:colocation-WebSite-WebFS-INFINITY)
>
> Cluster Properties:
> cluster-infrastructure: corosync
> dc-version: 1.1.10-19.el7-368c726
> last-lrm-refresh: 1388615115
> no-quorum-policy: ignore
> stonith-enabled: true
> Node Attributes:
> an-c03n01.alteeve.ca: standby=on
> =====
>
> Put an-c03n01 into unstandby:
>
> =====
> [root at an-c03n01 ~]# pcs cluster unstandby an-c03n01.alteeve.ca
>
> ==[an-c03n01 logs]===
> Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.191.7
> Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.192.1
> 9eac78791868c0b3878e59927a934eba
> Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: -- <nvpair
> id="nodes-1-standby" name="standby" value="on"/>
> Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: ++ <cib admin_epoch="0"
> cib-last-written="Wed Jan 1 18:22:32 2014" crm_feature_set="3.0.7" epoch="192"
> have-quorum="1" num_updates="1" update-client="crm_attribute"
> update-origin="an-c03n01.alteeve.ca" validate-with="pacemaker-1.2" dc-uuid="1"/>
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Move ClusterIP
> (Started an-c03n02.alteeve.ca -> an-c03n01.alteeve.ca)
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Start WebData:1
> (an-c03n01.alteeve.ca)
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Stop WebSite
> (an-c03n02.alteeve.ca)
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 42: /var/lib/pacemaker/pengine/pe-input-180.bz2
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 14: stop ClusterIP_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 49: stop WebSite_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 53: notify WebData_pre_notify_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 21: start WebData:1_start_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 15: start ClusterIP_start_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO: Adding inet address
> 192.168.122.10/32 to device eth0
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.282453] drbd r0: Starting worker thread
> (from drbdsetup [15705])
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283364] block drbd0: disk( Diskless ->
> Attaching )
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283429] drbd r0: Method to ensure write
> ordering: drain
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283431] block drbd0: max BIO size =
> 1048576
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283433] block drbd0: Adjusting my
> ra_pages to backing device's (32 -> 1024)
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283435] block drbd0: drbd_bm_resize
> called with capacity == 102392704
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283580] block drbd0: resync bitmap:
> bits=12799088 words=199986 pages=391
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283582] block drbd0: size = 49 GB
> (51196352 KB)
> Jan 1 18:22:32 an-c03n01 avahi-daemon[298]: Registering new address record for
> 192.168.122.10 on eth0.IPv4.
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285244] block drbd0: bitmap READ of 391
> pages took 2 jiffies
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285466] block drbd0: recounting of set
> bits took additional 0 jiffies
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285468] block drbd0: 0 KB (0 bits)
> marked out-of-sync by on disk bit-map.
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285473] block drbd0: disk( Attaching ->
> Consistent )
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285475] block drbd0: attached to UUIDs
> 6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
> Jan 1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO: Bringing device eth0 up
> Jan 1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO:
> /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
> /var/run/resource-agents/send_arp-192.168.122.10 eth0 192.168.122.10 auto
> not_used not_used
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292172] drbd r0: conn( StandAlone ->
> Unconnected )
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292185] drbd r0: Starting receiver
> thread (from drbd_w_r0 [15707])
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292199] drbd r0: receiver (re)started
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292205] drbd r0: conn( Unconnected ->
> WFConnection )
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> ClusterIP_start_0 (call=106, rc=0, cib-update=293, confirmed=true) ok
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 16: monitor ClusterIP_monitor_30000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:22:32 an-c03n01 attrd[1765]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-WebData (5)
> Jan 1 18:22:32 an-c03n01 attrd[1765]: notice: attrd_perform_update: Sent update
> 160: master-WebData=5
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_start_0 (call=105, rc=0, cib-update=294, confirmed=true) ok
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 54: notify WebData_post_notify_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 55: notify WebData:1_post_notify_start_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> ClusterIP_monitor_30000 (call=107, rc=0, cib-update=295, confirmed=false) ok
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=108, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793761] drbd r0: Handshake successful:
> Agreed network protocol version 101
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793765] drbd r0: Agreed to support TRIM
> on protocol level
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793809] drbd r0: conn( WFConnection ->
> WFReportParams )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793812] drbd r0: Starting asender
> thread (from drbd_r_r0 [15727])
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799253] block drbd0: drbd_sync_handshake:
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799258] block drbd0: self
> 6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC bits:0 flags:0
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799261] block drbd0: peer
> 3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC bits:1 flags:0
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799263] block drbd0: uuid_compare()=-1
> by rule 50
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799270] block drbd0: peer( Unknown ->
> Primary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent -> Outdated )
> pdsk( DUnknown -> UpToDate )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.803153] block drbd0: receive bitmap
> stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.803403] block drbd0: send bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.803409] block drbd0: conn( WFBitMapT ->
> WFSyncUUID )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.805842] block drbd0: updated sync uuid
> 6A9A4DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.806189] block drbd0: helper command:
> /sbin/drbdadm before-resync-target minor-0
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.807379] block drbd0: helper command:
> /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.807392] block drbd0: conn( WFSyncUUID
> -> SyncTarget ) disk( Outdated -> Inconsistent )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.807398] block drbd0: Began resync as
> SyncTarget (will sync 4 KB [1 bits set]).
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809259] block drbd0: Resync done (total
> 1 sec; paused 0 sec; 4 K/sec)
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809264] block drbd0: updated UUIDs
> 3E5C4A9426BEF114:0000000000000000:6A9A4DBF33FF45AE:6A994DBF33FF45AE
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809268] block drbd0: conn( SyncTarget
> -> Connected ) disk( Inconsistent -> UpToDate )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809382] block drbd0: helper command:
> /sbin/drbdadm after-resync-target minor-0
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.810777] block drbd0: helper command:
> /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 42
> (Complete=14, Pending=0, Fired=0, Skipped=2, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-180.bz2): Stopped
> Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 43: /var/lib/pacemaker/pengine/pe-input-181.bz2
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 17: monitor WebData_monitor_60000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:22:34 an-c03n01 attrd[1765]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-WebData (10000)
> Jan 1 18:22:34 an-c03n01 attrd[1765]: notice: attrd_perform_update: Sent update
> 162: master-WebData=10000
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_monitor_60000 (call=109, rc=0, cib-update=297, confirmed=false) ok
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 43
> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-181.bz2): Complete
> Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 44: /var/lib/pacemaker/pengine/pe-input-182.bz2
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 44
> (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-182.bz2): Complete
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
>
> ==[an-c03n02 logs]===
> Jan 1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=108, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:22:32 an-c03n02 apache(WebSite)[2631]: INFO: Attempting graceful stop
> of apache PID 32497
> Jan 1 18:22:32 an-c03n02 IPaddr2(ClusterIP)[2630]: INFO: IP status = ok, IP_CIP=
> Jan 1 18:22:32 an-c03n02 avahi-daemon[295]: Withdrawing address record for
> 192.168.122.10 on eth0.
> Jan 1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> ClusterIP_stop_0 (call=105, rc=0, cib-update=69, confirmed=true) ok
> Jan 1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=109, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:22:33 an-c03n02 kernel: [19832.801173] drbd r0: Handshake successful:
> Agreed network protocol version 101
> Jan 1 18:22:33 an-c03n02 kernel: [19832.801177] drbd r0: Agreed to support TRIM
> on protocol level
> Jan 1 18:22:33 an-c03n02 kernel: [19832.801209] drbd r0: conn( WFConnection ->
> WFReportParams )
> Jan 1 18:22:33 an-c03n02 kernel: [19832.801211] drbd r0: Starting asender
> thread (from drbd_r_r0 [19965])
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810255] block drbd0: drbd_sync_handshake:
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810260] block drbd0: self
> 3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC bits:1 flags:0
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810263] block drbd0: peer
> 6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC bits:0 flags:0
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810265] block drbd0: uuid_compare()=1
> by rule 70
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810270] block drbd0: peer( Unknown ->
> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated -> Consistent )
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810573] block drbd0: send bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
> Jan 1 18:22:33 an-c03n02 kernel: [19832.811274] block drbd0: receive bitmap
> stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
> Jan 1 18:22:33 an-c03n02 kernel: [19832.811279] block drbd0: helper command:
> /sbin/drbdadm before-resync-source minor-0
> Jan 1 18:22:33 an-c03n02 kernel: [19832.812937] block drbd0: helper command:
> /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
> Jan 1 18:22:33 an-c03n02 kernel: [19832.812951] block drbd0: conn( WFBitMapS ->
> SyncSource ) pdsk( Consistent -> Inconsistent )
> Jan 1 18:22:33 an-c03n02 kernel: [19832.812959] block drbd0: Began resync as
> SyncSource (will sync 4 KB [1 bits set]).
> Jan 1 18:22:33 an-c03n02 kernel: [19832.812996] block drbd0: updated sync UUID
> 3E5C4A9426BEF115:6A9A4DBF33FF45AE:6A994DBF33FF45AE:2A36647E5537B8FC
> Jan 1 18:22:33 an-c03n02 kernel: [19832.816963] block drbd0: Resync done (total
> 1 sec; paused 0 sec; 4 K/sec)
> Jan 1 18:22:33 an-c03n02 kernel: [19832.816978] block drbd0: updated UUIDs
> 3E5C4A9426BEF115:0000000000000000:6A9A4DBF33FF45AE:6A994DBF33FF45AE
> Jan 1 18:22:33 an-c03n02 kernel: [19832.816985] block drbd0: conn( SyncSource
> -> Connected ) pdsk( Inconsistent -> UpToDate )
> Jan 1 18:22:34 an-c03n02 apache(WebSite)[2631]: INFO: apache stopped.
> Jan 1 18:22:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebSite_stop_0 (call=107, rc=0, cib-update=70, confirmed=true) ok
> =====
>
> WebSite stopped and won't start now.
> =====
> [root at an-c03n01 ~]# pcs status
> Cluster name: an-cluster-03
> Last updated: Wed Jan 1 18:23:44 2014
> Last change: Wed Jan 1 18:22:32 2014 via crm_attribute on an-c03n01.alteeve.ca
> Stack: corosync
> Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
> Version: 1.1.10-19.el7-368c726
> 2 Nodes configured
> 7 Resources configured
>
>
> Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
>
> Full list of resources:
>
> fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n01.alteeve.ca
> Master/Slave Set: WebDataClone [WebData]
> Masters: [ an-c03n02.alteeve.ca ]
> Slaves: [ an-c03n01.alteeve.ca ]
> WebFS (ocf::heartbeat:Filesystem): Started an-c03n02.alteeve.ca
> WebSite (ocf::heartbeat:apache): Stopped
>
> PCSD Status:
> an-c03n01.alteeve.ca:
> an-c03n01.alteeve.ca: Online
> an-c03n02.alteeve.ca:
> an-c03n02.alteeve.ca: Online
>
> Daemon Status:
> corosync: active/disabled
> pacemaker: active/disabled
> pcsd: active/enabled
> =====
>
> Here is what I see when I try to start it manually:
>
> =====
> [root at an-c03n01 ~]# pcs resource debug-start WebSite --full
> Operation start for WebSite (ocf:heartbeat:apache) returned 0
> > stderr: + 18:24:17: 46: . /usr/lib/ocf/lib/heartbeat/apache-conf.sh
> > stderr: + 18:24:17: 47: . /usr/lib/ocf/lib/heartbeat/http-mon.sh
> > stderr: ++ 18:24:17: 18: bind_address=127.0.0.1
> > stderr: ++ 18:24:17: 19: curl_ipv6_opts=
> > stderr: ++ 18:24:17: 20: ocf_is_true ''
> > stderr: ++ 18:24:17: ocf_is_true:101: case "$1" in
> > stderr: ++ 18:24:17: ocf_is_true:103: false
> > stderr: ++ 18:24:17: 20: grep -qs ::
> > stderr: ++ 18:24:17: 20: echo ''
> > stderr: ++ 18:24:17: 24: WGETOPTS='-O- -q -L --no-proxy
> --bind-address=127.0.0.1'
> > stderr: ++ 18:24:17: 25: CURLOPTS='-o - -Ss -L --interface lo '
> > stderr: + 18:24:17: 48: HA_VARRUNDIR=/var/run/
> > stderr: + 18:24:17: 56: IBMHTTPD=/opt/IBMHTTPServer/bin/httpd
> > stderr: + 18:24:17: 57: HTTPDLIST='/sbin/httpd2 /usr/sbin/httpd2
> /usr/sbin/apache2 /sbin/httpd /usr/sbin/httpd /usr/sbin/apache
> /opt/IBMHTTPServer/bin/httpd'
> > stderr: + 18:24:17: 58: MPM=/usr/share/apache2/find_mpm
> > stderr: + 18:24:17: 59: '[' -x /usr/share/apache2/find_mpm ']'
> > stderr: + 18:24:17: 63: LOCALHOST=http://localhost
> > stderr: + 18:24:17: 64: HTTPDOPTS=-DSTATUS
> > stderr: + 18:24:17: 65: DEFAULT_IBMCONFIG=/opt/IBMHTTPServer/conf/httpd.conf
> > stderr: + 18:24:17: 66: DEFAULT_SUSECONFIG=/etc/apache2/httpd.conf
> > stderr: + 18:24:17: 67: DEFAULT_RHELCONFIG=/etc/httpd/conf/httpd.conf
> > stderr: ++ 18:24:17: 79: basename /usr/lib/ocf/resource.d/heartbeat/apache
> > stderr: + 18:24:17: 79: CMD=apache
> > stderr: + 18:24:17: 654: OCF_REQUIRED_PARAMS=
> > stderr: + 18:24:17: 655: OCF_REQUIRED_BINARIES=
> > stderr: + 18:24:17: 656: ocf_rarun start
> > stderr: + 18:24:17: ocf_rarun:137: mk_action_func
> > stderr: ++ 18:24:17: mk_action_func:50: echo apache_start
> > stderr: ++ 18:24:17: mk_action_func:50: tr - _
> > stderr: + 18:24:17: mk_action_func:50: ACTION_FUNC=apache_start
> > stderr: + 18:24:17: ocf_rarun:138: validate_args
> > stderr: + 18:24:17: validate_args:53: is_function apache_start
> > stderr: ++ 18:24:17: is_function:41: command -v apache_start
> > stderr: + 18:24:17: is_function:41: test zapache_start = zapache_start
> > stderr: + 18:24:17: ocf_rarun:139: simple_actions
> > stderr: + 18:24:17: simple_actions:60: case $__OCF_ACTION in
> > stderr: + 18:24:17: ocf_rarun:140: check_required_params
> > stderr: + 18:24:17: check_required_params:74: local v
> > stderr: + 18:24:17: ocf_rarun:141: run_function apache_getconfig
> > stderr: + 18:24:17: run_function:44: is_function apache_getconfig
> > stderr: ++ 18:24:17: is_function:41: command -v apache_getconfig
> > stderr: + 18:24:17: is_function:41: test zapache_getconfig = zapache_getconfig
> > stderr: + 18:24:17: run_function:44: apache_getconfig
> > stderr: + 18:24:17: apache_getconfig:628: HTTPD=
> > stderr: + 18:24:17: apache_getconfig:629: PORT=
> > stderr: + 18:24:17: apache_getconfig:630:
> STATUSURL=http://127.0.0.1/server-status
> > stderr: + 18:24:17: apache_getconfig:631:
> CONFIGFILE=/etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: apache_getconfig:632: OPTIONS=
> > stderr: + 18:24:17: apache_getconfig:633: CLIENT=
> > stderr: + 18:24:17: apache_getconfig:634: TESTREGEX='</ *html *>'
> > stderr: + 18:24:17: apache_getconfig:635: TESTURL=
> > stderr: + 18:24:17: apache_getconfig:636: TESTREGEX10=
> > stderr: + 18:24:17: apache_getconfig:637: TESTCONFFILE=
> > stderr: + 18:24:17: apache_getconfig:638: TESTNAME=
> > stderr: + 18:24:17: apache_getconfig:639: : /etc/apache2/envvars
> > stderr: + 18:24:17: apache_getconfig:640: source_envfiles /etc/apache2/envvars
> > stderr: + 18:24:17: source_envfiles:16: for f in '"$@"'
> > stderr: + 18:24:17: source_envfiles:17: '[' -f /etc/apache2/envvars -a -r
> /etc/apache2/envvars ']'
> > stderr: + 18:24:17: apache_getconfig:642: '[' X = X -o '!' -f '' -o '!' -x
> '' ']'
> > stderr: + 18:24:17: apache_getconfig:643: find_httpd_prog
> > stderr: + 18:24:17: find_httpd_prog:603: case $0 in
> > stderr: + 18:24:17: find_httpd_prog:608: HTTPD=
> > stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> > stderr: + 18:24:17: find_httpd_prog:611: '[' -f /sbin/httpd2 -a -x
> /sbin/httpd2 ']'
> > stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> > stderr: + 18:24:17: find_httpd_prog:611: '[' -f /usr/sbin/httpd2 -a -x
> /usr/sbin/httpd2 ']'
> > stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> > stderr: + 18:24:17: find_httpd_prog:611: '[' -f /usr/sbin/apache2 -a -x
> /usr/sbin/apache2 ']'
> > stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> > stderr: + 18:24:17: find_httpd_prog:611: '[' -f /sbin/httpd -a -x
> /sbin/httpd ']'
> > stderr: + 18:24:17: find_httpd_prog:612: HTTPD=/sbin/httpd
> > stderr: + 18:24:17: find_httpd_prog:613: break
> > stderr: + 18:24:17: find_httpd_prog:618: '[' X '!=' X -a X/sbin/httpd '!='
> X ']'
> > stderr: ++ 18:24:17: find_httpd_prog:621: detect_default_config
> > stderr: ++ 18:24:17: detect_default_config:387: '[' -f
> /etc/apache2/httpd.conf ']'
> > stderr: ++ 18:24:17: detect_default_config:390: echo
> /etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: find_httpd_prog:621:
> DefaultConfig=/etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: apache_getconfig:646:
> CONFIGFILE=/etc/httpd/conf/httpd.conf
> > stderr: ++ 18:24:17: apache_getconfig:647: basename /sbin/httpd
> > stderr: + 18:24:17: apache_getconfig:647: httpd_basename=httpd
> > stderr: + 18:24:17: apache_getconfig:648: case $httpd_basename in
> > stderr: + 18:24:17: apache_getconfig:651: GetParams /etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: GetParams:134: ConfigFile=/etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: GetParams:135: '[' '!' -f /etc/httpd/conf/httpd.conf ']'
> > stderr: + 18:24:17: GetParams:138: get_apache_params
> /etc/httpd/conf/httpd.conf ServerRoot PidFile Port Listen
> > stderr: + 18:24:17: get_apache_params:79:
> configfile=/etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: get_apache_params:80: shift 1
> > stderr: ++ 18:24:17: get_apache_params:81: echo ServerRoot PidFile Port Listen
> > stderr: ++ 18:24:17: get_apache_params:81: sed 's/ /,/g'
> > stderr: + 18:24:17: get_apache_params:81: vars=ServerRoot,PidFile,Port,Listen
> > stderr: ++ 18:24:17: get_apache_params:99: apachecat
> /etc/httpd/conf/httpd.conf
> > stderr: ++ 18:24:17: get_apache_params:99: awk -v
> vars=ServerRoot,PidFile,Port,Listen '
> > stderr: BEGIN{
> > stderr: split(vars,v,",");
> > stderr: for( i in v )
> > stderr: vl[i]=tolower(v[i]);
> > stderr: }
> > stderr: {
> > stderr: for( i in v )
> > stderr: if( tolower($1)==vl[i] ) {
> > stderr: print v[i]"="$2
> > stderr: delete vl[i]
> > stderr: break
> > stderr: }
> > stderr: }
> > stderr: '
> > stderr: ++ 18:24:17: apachecat:71: sed
> 's/#.*//;s/[[:blank:]]*$//;s/^[[:blank:]]*//'
> > stderr: ++ 18:24:17: apachecat:70: awk '
> > stderr: function procline() {
> > stderr: split($0,a);
> > stderr: if( a[1]~/^[Ii]nclude$/ ) {
> > stderr: procinclude(a[2]);
> > stderr: } else {
> > stderr: if( a[1]=="ServerRoot" ) {
> > stderr: rootdir=a[2];
> > stderr: gsub("\"","",rootdir);
> > stderr: }
> > stderr: print;
> > stderr: }
> > stderr: }
> > stderr: function printfile(infile, a) {
> > stderr: while( (getline<infile) > 0 ) {
> > stderr: procline();
> > stderr: }
> > stderr: close(infile);
> > stderr: }
> > stderr: function allfiles(dir, cmd,f) {
> > stderr: cmd="find -L "dir" -type f";
> > stderr: while( ( cmd | getline f ) > 0 ) {
> > stderr: printfile(f);
> > stderr: }
> > stderr: close(cmd);
> > stderr: }
> > stderr: function listfiles(pattern, cmd,f) {
> > stderr: cmd="ls "pattern" 2>/dev/null";
> > stderr: while( ( cmd | getline f ) > 0 ) {
> > stderr: printfile(f);
> > stderr: }
> > stderr: close(cmd);
> > stderr: }
> > stderr: function procinclude(spec) {
> > stderr: if( rootdir!="" && spec!~/^\// ) {
> > stderr: spec=rootdir"/"spec;
> > stderr: }
> > stderr: if( isdir(spec) ) {
> > stderr: allfiles(spec); # read all files in a directory (and
> subdirs)
> > stderr: } else {
> > stderr: listfiles(spec); # there could be jokers
> > stderr: }
> > stderr: }
> > stderr: function isdir(s) {
> > stderr: return !system("test -d \""s"\"");
> > stderr: }
> > stderr: { procline(); }
> > stderr: ' /etc/httpd/conf/httpd.conf
> > stderr: ++ 18:24:17: apachecat:72: grep -v '^$'
> > stderr: + 18:24:17: get_apache_params:98: eval 'ServerRoot="/etc/httpd"'
> Listen=80
> > stderr: ++ 18:24:17: get_apache_params:98: ServerRoot=/etc/httpd
> > stderr: ++ 18:24:17: get_apache_params:98: Listen=80
> > stderr: + 18:24:17: GetParams:139: case $PidFile in
> > stderr: + 18:24:17: GetParams:145: PidFile=/var/run//httpd.pid
> > stderr: + 18:24:17: GetParams:149: PIDFILE_DIRECTIVE=true
> > stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
> > stderr: + 18:24:17: GetParams:154: CheckPort ''
> > stderr: + 18:24:17: CheckPort:116: ocf_is_decimal ''
> > stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: + 18:24:17: ocf_is_decimal:94: false
> > stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
> > stderr: + 18:24:17: GetParams:154: CheckPort ''
> > stderr: + 18:24:17: CheckPort:116: ocf_is_decimal ''
> > stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: + 18:24:17: ocf_is_decimal:94: false
> > stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
> > stderr: + 18:24:17: GetParams:154: CheckPort 80
> > stderr: + 18:24:17: CheckPort:116: ocf_is_decimal 80
> > stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: + 18:24:17: ocf_is_decimal:96: true
> > stderr: + 18:24:17: CheckPort:116: '[' 80 -gt 0 ']'
> > stderr: + 18:24:17: GetParams:155: PORT=80
> > stderr: + 18:24:17: GetParams:156: break
> > stderr: + 18:24:17: GetParams:160: echo 80
> > stderr: + 18:24:17: GetParams:160: grep :
> > stderr: + 18:24:17: GetParams:161: Listen=localhost:80
> > stderr: + 18:24:17: GetParams:183: '[' Xhttp://127.0.0.1/server-status = X ']'
> > stderr: + 18:24:17: GetParams:189: test /var/run//httpd.pid
> > stderr: + 18:24:17: GetParams:192: return 0
> > stderr: + 18:24:17: ocf_rarun:142: validate_env
> > stderr: + 18:24:17: validate_env:122: check_required_binaries
> > stderr: + 18:24:17: check_required_binaries:113: local v
> > stderr: + 18:24:17: validate_env:123: is_function apache_validate_all
> > stderr: ++ 18:24:17: is_function:41: command -v apache_validate_all
> > stderr: + 18:24:17: is_function:41: test zapache_validate_all =
> zapache_validate_all
> > stderr: + 18:24:17: validate_env:125: local rc
> > stderr: + 18:24:17: validate_env:126: LSB_STATUS_STOPPED=3
> > stderr: + 18:24:17: validate_env:127: apache_validate_all
> > stderr: + 18:24:17: apache_validate_all:573: CheckPort 80
> > stderr: + 18:24:17: CheckPort:116: ocf_is_decimal 80
> > stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: + 18:24:17: ocf_is_decimal:96: true
> > stderr: + 18:24:17: CheckPort:116: '[' 80 -gt 0 ']'
> > stderr: + 18:24:17: apache_validate_all:575: : OK
> > stderr: + 18:24:17: apache_validate_all:581: case $STATUSURL in
> > stderr: + 18:24:17: apache_validate_all:588: '[' '!' -x /sbin/httpd ']'
> > stderr: + 18:24:17: apache_validate_all:593: '[' '!' -f
> /etc/httpd/conf/httpd.conf ']'
> > stderr: + 18:24:17: apache_validate_all:599: return 0
> > stderr: + 18:24:17: validate_env:128: rc=0
> > stderr: + 18:24:17: validate_env:129: '[' 0 -ne 0 ']'
> > stderr: + 18:24:17: ocf_rarun:143: ocf_is_probe
> > stderr: + 18:24:17: ocf_is_probe:438: '[' start = monitor -a 0 = 0 ']'
> > stderr: + 18:24:17: ocf_rarun:144: shift 1
> > stderr: + 18:24:17: ocf_rarun:145: apache_start
> > stderr: + 18:24:17: apache_start:164: silent_status
> > stderr: + 18:24:17: silent_status:125: '[' -f /var/run//httpd.pid ']'
> > stderr: + 18:24:17: silent_status:128: : No pid file
> > stderr: + 18:24:17: silent_status:129: false
> > stderr: + 18:24:17: apache_start:170: validate_default_config
> > stderr: + 18:24:17: validate_default_config:135: '[' -e /etc/SuSE-release ']'
> > stderr: + 18:24:17: validate_default_config:138: return 0
> > stderr: + 18:24:17: apache_start:172: '[' -d /var/run/apache2 ']'
> > stderr: + 18:24:17: apache_start:174: '[' -z true ']'
> > stderr: + 18:24:17: apache_start:177: ocf_run /sbin/httpd -DSTATUS -f
> /etc/httpd/conf/httpd.conf -c 'PidFile /var/run//httpd.pid'
> > stderr: + 18:24:17: ocf_run:361: local rc
> > stderr: + 18:24:17: ocf_run:362: local output
> > stderr: + 18:24:17: ocf_run:363: local verbose=1
> > stderr: + 18:24:17: ocf_run:364: local loglevel=err
> > stderr: + 18:24:17: ocf_run:365: local var
> > stderr: + 18:24:17: ocf_run:367: for var in 1 2
> > stderr: + 18:24:17: ocf_run:369: case "$1" in
> > stderr: + 18:24:17: ocf_run:367: for var in 1 2
> > stderr: + 18:24:17: ocf_run:369: case "$1" in
> > stderr: ++ 18:24:17: ocf_run:381: /sbin/httpd -DSTATUS -f
> /etc/httpd/conf/httpd.conf -c 'PidFile /var/run//httpd.pid'
> > stderr: + 18:24:17: ocf_run:381: output=
> > stderr: + 18:24:17: ocf_run:382: rc=0
> > stderr: ++ 18:24:17: ocf_run:383: echo
> > stderr: + 18:24:17: ocf_run:383: output=
> > stderr: + 18:24:17: ocf_run:384: '[' 0 -eq 0 ']'
> > stderr: + 18:24:17: ocf_run:385: '[' 1 -a '!' -z '' ']'
> > stderr: + 18:24:17: ocf_run:388: return 0
> > stderr: + 18:24:17: apache_start:180: tries=0
> > stderr: + 18:24:17: apache_start:181: :
> > stderr: + 18:24:17: apache_start:183: apache_monitor
> > stderr: + 18:24:17: apache_monitor:367: silent_status
> > stderr: + 18:24:17: silent_status:125: '[' -f /var/run//httpd.pid ']'
> > stderr: ++ 18:24:17: silent_status:126: cat /var/run//httpd.pid
> > stderr: + 18:24:17: silent_status:126: ProcessRunning 16404
> > stderr: + 18:24:17: ProcessRunning:114: ApachePID=16404
> > stderr: + 18:24:17: ProcessRunning:116: '[' -d /proc -a -d /proc/1 ']'
> > stderr: + 18:24:17: ProcessRunning:117: '[' -d /proc/16404 ']'
> > stderr: + 18:24:17: apache_monitor:368: '[' 0 -ne 0 ']'
> > stderr: ++ 18:24:17: apache_monitor:373: findhttpclient
> > stderr: ++ 18:24:17: findhttpclient:66: '[' x '!=' x ']'
> > stderr: ++ 18:24:17: findhttpclient:68: which wget
> > stderr: ++ 18:24:17: findhttpclient:69: echo wget
> > stderr: + 18:24:17: apache_monitor:373: ourhttpclient=wget
> > stderr: + 18:24:17: apache_monitor:374: '[' -z wget ']'
> > stderr: + 18:24:17: apache_monitor:379: case `ocf_check_level 10` in
> > stderr: ++ 18:24:17: apache_monitor:379: ocf_check_level 10
> > stderr: ++ 18:24:17: ocf_check_level:643: local lvl prev
> > stderr: ++ 18:24:17: ocf_check_level:644: lvl=0
> > stderr: ++ 18:24:17: ocf_check_level:645: prev=0
> > stderr: ++ 18:24:17: ocf_check_level:646: ocf_is_decimal 0
> > stderr: ++ 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: ++ 18:24:17: ocf_is_decimal:96: true
> > stderr: ++ 18:24:17: ocf_check_level:648: for lvl in '"$@"'
> > stderr: ++ 18:24:17: ocf_check_level:649: '[' 10 -eq 0 ']'
> > stderr: ++ 18:24:17: ocf_check_level:651: '[' 10 -gt 0 ']'
> > stderr: ++ 18:24:17: ocf_check_level:652: lvl=0
> > stderr: ++ 18:24:17: ocf_check_level:653: break
> > stderr: ++ 18:24:17: ocf_check_level:658: echo 0
> > stderr: + 18:24:17: apache_monitor:380: apache_monitor_basic
> > stderr: + 18:24:17: apache_monitor_basic:351: wget_func
> http://127.0.0.1/server-status
> > stderr: + 18:24:17: apache_monitor_basic:351: grep -Ei '</ *html *>'
> > stderr: + 18:24:17: wget_func:49: auth=
> > stderr: + 18:24:17: wget_func:50: cl_opts='-O- -q -L --no-proxy
> --bind-address=127.0.0.1 '
> > stderr: + 18:24:17: wget_func:51: '[' x '!=' x ']'
> > stderr: + 18:24:17: wget_func:53: wget -O- -q -L --no-proxy
> --bind-address=127.0.0.1 http://127.0.0.1/server-status
> > stderr: + 18:24:17: apache_monitor_basic:353: return 0
> > stderr: + 18:24:17: apache_start:184: ec=0
> > stderr: + 18:24:17: apache_start:185: '[' 0 -eq 7 ']'
> > stderr: + 18:24:17: apache_start:191: break
> > stderr: + 18:24:17: apache_start:195: '[' 0 -ne 0 ']'
> > stderr: + 18:24:17: apache_start:198: return 0
> =====
>
> Only log entry is on an-c03n01 (where I ran the command):
>
> =====
> Jan 1 18:24:17 an-c03n01 crm_resource[16170]: notice: crm_log_args: Invoked:
> /usr/sbin/crm_resource -r WebSite --force-start -V
> =====
>
> When I delete the constraint that was added when the node went into standby,
> WebSite starts.
>
> =====
> [root at an-c03n01 ~]# pcs config show | grep uname -B 1
> Rule: score=-INFINITY role=Master
> (id:drbd-fence-by-handler-r0-rule-WebDataClone)
> Expression: #uname ne an-c03n02.alteeve.ca
> (id:drbd-fence-by-handler-r0-expr-WebDataClone)
> [root at an-c03n01 ~]# pcs constraint delete
> drbd-fence-by-handler-r0-rule-WebDataClone
> =====
>
> ==[an-c03n01 logs]===
> Jan 1 18:26:44 an-c03n01 cibadmin[17037]: notice: crm_log_args: Invoked:
> /usr/sbin/cibadmin -c -R --xml-text <constraints>
> <rsc_location id="cli-prefer-WebDataClone" node="an-c03n01.alteeve.ca"
> role="Started" rsc="WebDataClone" score="INFINITY"/>
> <rsc_location id="cli-prefer-ClusterIP" node="an-c03n01.alteeve.ca"
> role="Started" rsc="ClusterIP" score="INFINITY"/>
> <rsc_colocation id="colocation-WebFS-WebDataClone-INFINITY" rsc="WebFS"
> score="INFINITY" with-rsc="WebDataClone" with-rsc-role="Master"/>
> <rsc_order first="WebDataClone" first-ac
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.192.9
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.193.1
> 695289e37618e5e9a184d2bc8c90d421
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- <rsc_location
> rsc="WebDataClone" id="drbd-fence-by-handler-r0-WebDataClone">
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- <rule
> role="Master" score="-INFINITY" id="drbd-fence-by-handler-r0-rule-WebDataClone">
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- <expression
> attribute="#uname" operation="ne" value="an-c03n02.alteeve.ca"
> id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- </rule>
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- </rsc_location>
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: ++ <cib admin_epoch="0"
> cib-last-written="Wed Jan 1 18:26:44 2014" crm_feature_set="3.0.7" epoch="193"
> have-quorum="1" num_updates="1" update-client="cibadmin"
> update-origin="an-c03n01.alteeve.ca" validate-with="pacemaker-1.2" dc-uuid="1"/>
> Jan 1 18:26:44 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
> Jan 1 18:26:44 an-c03n01 stonith-ng[1763]: notice: unpack_config: On loss of
> CCM Quorum: Ignore
> Jan 1 18:26:44 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_FSA_INTERNAL origin=do_election_check ]
> Jan 1 18:26:44 an-c03n01 stonith-ng[1763]: notice: stonith_device_register:
> Added 'fence_n01_virsh' to the device list (2 active devices)
> Jan 1 18:26:44 an-c03n01 stonith-ng[1763]: notice: stonith_device_register:
> Added 'fence_n02_virsh' to the device list (2 active devices)
> Jan 1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_local_callback: Sending
> full refresh (origin=crmd)
> Jan 1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-WebData (10000)
> Jan 1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: probe_complete (true)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:0 (Slave -> Master an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Demote WebData:1
> (Master -> Slave an-c03n02.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Move WebFS
> (Started an-c03n02.alteeve.ca -> an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start WebSite
> (an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 45: /var/lib/pacemaker/pengine/pe-input-183.bz2
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 2: cancel WebData_cancel_60000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 47: stop WebFS_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 67: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 69: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=111, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: run_graph: Transition 45
> (Complete=5, Pending=0, Fired=0, Skipped=15, Incomplete=10,
> Source=/var/lib/pacemaker/pengine/pe-input-183.bz2): Stopped
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:0 (Slave -> Master an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Demote WebData:1
> (Master -> Slave an-c03n02.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start WebFS
> (an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start WebSite
> (an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: process_pe_message: Calculated
> Transition 46: /var/lib/pacemaker/pengine/pe-input-184.bz2
> Jan 1 18:26:46 an-c03n01 crmd[1767]: warning: destroy_action: Cancelling timer
> for action 2 (src=476)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 64: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 66: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=112, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 18: demote WebData_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 kernel: [ 3524.517000] block drbd0: peer( Primary ->
> Secondary )
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 65: notify WebData_post_notify_demote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 67: notify WebData_post_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=113, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 60: notify WebData_pre_notify_promote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 62: notify WebData_pre_notify_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=114, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 16: promote WebData_promote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 kernel: [ 3524.605607] block drbd0: role( Secondary ->
> Primary )
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_promote_0 (call=115, rc=0, cib-update=324, confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 61: notify WebData_post_notify_promote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 63: notify WebData_post_notify_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=116, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 45: start WebFS_start_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 20: monitor WebData_monitor_60000 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 Filesystem(WebFS)[17211]: INFO: Running start for
> /dev/drbd0 on /var/www/html
> Jan 1 18:26:46 an-c03n01 kernel: [ 3524.688633] EXT4-fs (drbd0): mounted
> filesystem with ordered data mode. Opts: (null)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebFS_start_0 (call=117, rc=0, cib-update=325, confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 46: monitor WebFS_monitor_60000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 47: start WebSite_start_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 apache(WebSite)[17276]: INFO: apache already running
> (pid 16404)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebSite_start_0 (call=119, rc=0, cib-update=326, confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebFS_monitor_60000 (call=118, rc=0, cib-update=327, confirmed=false) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating action
> 48: monitor WebSite_monitor_60000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM operation
> WebSite_monitor_60000 (call=120, rc=0, cib-update=328, confirmed=false) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: run_graph: Transition 46
> (Complete=27, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-184.bz2): Complete
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
>
> ==[an-c03n02 logs]===
> Jan 1 18:26:44 an-c03n02 crmd[16388]: notice: do_state_transition: State
> transition S_NOT_DC -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
> origin=do_election_count_vote ]
> Jan 1 18:26:44 an-c03n02 stonith-ng[16384]: notice: unpack_config: On loss of
> CCM Quorum: Ignore
> Jan 1 18:26:44 an-c03n02 stonith-ng[16384]: notice: stonith_device_register:
> Device 'fence_n01_virsh' already existed in device list (2 active devices)
> Jan 1 18:26:44 an-c03n02 stonith-ng[16384]: notice: stonith_device_register:
> Device 'fence_n02_virsh' already existed in device list (2 active devices)
> Jan 1 18:26:45 an-c03n02 crmd[16388]: notice: do_state_transition: State
> transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE
> origin=do_cl_join_finalize_respond ]
> Jan 1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_local_callback: Sending
> full refresh (origin=crmd)
> Jan 1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-WebData (10000)
> Jan 1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: probe_complete (true)
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=112, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: Running stop for
> /dev/drbd0 on /var/www/html
> Jan 1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: Trying to unmount
> /var/www/html
> Jan 1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: unmounted /var/www/html
> successfully
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebFS_stop_0 (call=111, rc=0, cib-update=73, confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=113, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 kernel: [20086.520840] block drbd0: role( Primary ->
> Secondary )
> Jan 1 18:26:46 an-c03n02 kernel: [20086.520911] block drbd0: bitmap WRITE of 0
> pages took 0 jiffies
> Jan 1 18:26:46 an-c03n02 kernel: [20086.520914] block drbd0: 0 KB (0 bits)
> marked out-of-sync by on disk bit-map.
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_demote_0 (call=114, rc=0, cib-update=74, confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=115, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=116, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 kernel: [20086.610152] block drbd0: peer( Secondary ->
> Primary )
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_notify_0 (call=117, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM operation
> WebData_monitor_60000 (call=118, rc=0, cib-update=75, confirmed=false) ok
> =====
>
> Now WebSite is working:
> =====
> [root at an-c03n01 ~]# pcs status
> Cluster name: an-cluster-03
> Last updated: Wed Jan 1 18:28:04 2014
> Last change: Wed Jan 1 18:26:44 2014 via cibadmin on an-c03n01.alteeve.ca
> Stack: corosync
> Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
> Version: 1.1.10-19.el7-368c726
> 2 Nodes configured
> 7 Resources configured
>
>
> Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
>
> Full list of resources:
>
> fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n01.alteeve.ca
> Master/Slave Set: WebDataClone [WebData]
> Masters: [ an-c03n01.alteeve.ca ]
> Slaves: [ an-c03n02.alteeve.ca ]
> WebFS (ocf::heartbeat:Filesystem): Started an-c03n01.alteeve.ca
> WebSite (ocf::heartbeat:apache): Started an-c03n01.alteeve.ca
>
> PCSD Status:
> an-c03n01.alteeve.ca:
> an-c03n01.alteeve.ca: Online
> an-c03n02.alteeve.ca:
> an-c03n02.alteeve.ca: Online
>
> Daemon Status:
> corosync: active/disabled
> pacemaker: active/disabled
> pcsd: active/enabled
> =====
>
> Is this a bug?
>
More information about the Pacemaker
mailing list