[Pacemaker] Problem with failover/failback under Ubuntu 10.04 for Active/Passive OpenNMS
Dan Frincu
dfrincu at streamwide.ro
Mon Jul 5 11:40:55 UTC 2010
> 6. Problem with failover/failback under Ubuntu 10.04 for
> Active/Passive OpenNMS (Sven Wick)
>
>
> Hi,
>
> I am trying to build an Active/Passive OpenNMS installation
> but have problems with failover and failback.
> Doing all manually (DRBD, VIP, Filesystem, ...) works just fine.
>
> I read many tutorials, the official book and the FAQ
> but am still stuck with my problem.
>
> I use Ubuntu 10.04 with "pacemaker" and "corosync".
> Hosts are 2 HP ProLiant G6 and are connected through a Cisco Switch.
> The switch was configured to allow Multicast as described in[1].
>
> Hostnames are "monitoring-node-01" and "monitoring-node-02"
> and it seems I can failover to "monitoring-node-02" but not back.
>
> The DRBD Init Script is disabled on both nodes.
>
> First my current config:
>
> crm configure show
>
> node monitoring-node-01 \
> attributes standby="off"
> node monitoring-node-02 \
> attributes standby="off"
> primitive drbd-opennms-config ocf:linbit:drbd \
> params drbd_resource="config" \
> op start interval="0" timeout="300s" \
> op stop interval="0" timeout="300s"
> primitive drbd-opennms-data ocf:linbit:drbd \
> params drbd_resource="data" \
> op start interval="0" timeout="300s" \
> op stop interval="0" timeout="300s"
> primitive drbd-opennms-db ocf:linbit:drbd \
> params drbd_resource="db" \
> op start interval="0" timeout="300s" \
> op stop interval="0" timeout="300s"
> primitive fs-opennms-config ocf:heartbeat:Filesystem \
> params device="/dev/drbd/by-res/config" directory="/etc/opennms"
> fstype="xfs" \
> op stop interval="0" timeout="300s" \
> op start interval="0" timeout="300s"
> primitive fs-opennms-data ocf:heartbeat:Filesystem \
> params device="/dev/drbd/by-res/data" directory="/var/lib/opennms"
> fstype="xfs" \
> op stop interval="0" timeout="300s" \
> op start interval="0" timeout="300s"
> primitive fs-opennms-db ocf:heartbeat:Filesystem \
> params device="/dev/drbd/by-res/db" directory="/var/lib/postgresql"
> fstype="xfs" \
> op stop interval="0" timeout="300s" \
> op start interval="0" timeout="300s"
> primitive opennms lsb:opennms \
> op start interval="0" timeout="300s" \
> op stop interval="0" timeout="300s" \
> meta target-role="Started"
> primitive postgres lsb:postgresql-8.4
> primitive vip ocf:heartbeat:IPaddr2 \
> params ip="172.24.25.20" cidr_netmask="24" nic="bond0"
> group dependencies vip fs-opennms-config fs-opennms-db fs-opennms-data
> postgres opennms
> ms ms-opennms-config drbd-opennms-config \
> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"
> notify="true"
> ms ms-opennms-data drbd-opennms-data \
> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"
> notify="true"
> ms ms-opennms-db drbd-opennms-db \
> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"
> notify="true"
> order promote-before-drbd-config inf: ms-opennms-config:promote
> fs-opennms-config:start
> order promote-before-drbd-data inf: ms-opennms-data:promote
> fs-opennms-data:start
> order promote-before-drbd-db inf: ms-opennms-db:promote
> fs-opennms-db:start
> property $id="cib-bootstrap-options" \
> dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="2" \
> stonith-enabled="false" \
> no-quorum-policy="ignore" \
> last-lrm-refresh="1278329430"
> rsc_defaults $id="rsc-options" \
> resource-stickiness="100"
>
>
> Current state if all works:
>
>
> monitoring-node-01
>
>
> cat /proc/drbd
>
> version: 8.3.7 (api:88/proto:86-91)
> GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
> root at monitoring-node-01, 2010-06-22 20:00:41
> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
> ns:0 nr:0 dw:2052 dr:29542 al:2 bm:8 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
> oos:0
> 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
> ns:0 nr:0 dw:0 dr:200 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
> 2: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r----
> ns:0 nr:0 dw:0 dr:200 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
>
> df -h
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/cciss/c0d0p1 270G 3.1G 253G 2% /
> none 16G 212K 16G 1% /dev
> none 16G 3.5M 16G 1% /dev/shm
> none 16G 84K 16G 1% /var/run
> none 16G 12K 16G 1% /var/lock
> none 16G 0 16G 0% /lib/init/rw
> /dev/drbd0 10G 34M 10G 1% /etc/opennms
> /dev/drbd1 50G 67M 50G 1% /var/lib/postgresql
> /dev/drbd2 214G 178M 214G 1% /var/lib/opennms
>
>
> crm_mon
>
> ============
> Last updated: Mon Jul 5 13:02:54 2010
> Stack: openais
> Current DC: monitoring-node-02 - partition with quorum
> Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
> 2 Nodes configured, 2 expected votes
> 4 Resources configured.
> ============
>
> Online: [ monitoring-node-01 monitoring-node-02 ]
>
> Master/Slave Set: ms-opennms-data
> Masters: [ monitoring-node-01 ]
> Slaves: [ monitoring-node-02 ]
> Master/Slave Set: ms-opennms-db
> Masters: [ monitoring-node-01 ]
> Slaves: [ monitoring-node-02 ]
> Master/Slave Set: ms-opennms-config
> Masters: [ monitoring-node-01 ]
> Slaves: [ monitoring-node-02 ]
> Resource Group: dependencies
> vip (ocf::heartbeat:IPaddr2): Started monitoring-node-01
> fs-opennms-config (ocf::heartbeat:Filesystem): Started
> monitoring-node-01
> fs-opennms-db (ocf::heartbeat:Filesystem): Started
> monitoring-node-01
> fs-opennms-data (ocf::heartbeat:Filesystem): Started
> monitoring-node-01
> postgres (lsb:postgresql-8.4): Started monitoring-node-01
> opennms (lsb:opennms): Started monitoring-node-01
>
>
>
> monitoring-node-02
>
> cat /proc/drbd
>
> version: 8.3.7 (api:88/proto:86-91)
> GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
> root at monitoring-node-02, 2010-06-22 19:59:35
> 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
> ns:0 nr:0 dw:24620 dr:400 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
> oos:0
> 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
> ns:0 nr:302898 dw:302898 dr:400 al:0 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1
> wo:d oos:0
> 2: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r----
> ns:0 nr:19712467 dw:19712467 dr:400 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1
> wo:d oos:0
>
> df -h
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/cciss/c0d0p1 270G 1.8G 254G 1% /
> none 16G 212K 16G 1% /dev
> none 16G 3.6M 16G 1% /dev/shm
> none 16G 72K 16G 1% /var/run
> none 16G 12K 16G 1% /var/lock
> none 16G 0 16G 0% /lib/init/rw
>
>
>
>
> Rebooting "monitoring-node01" seems to work. All services run now
> on "monitoring-node02" and were started in correct order.
>
> But when I reboot "monitoring-node02":
>
> crm_mon
>
> Last updated: Mon Jul 5 13:21:31 2010
> Stack: openais
> Current DC: monitoring-node-01 - partition with quorum
> Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
> 2 Nodes configured, 2 expected votes
> 4 Resources configured.
> ============
>
> Online: [ monitoring-node-01 monitoring-node-02 ]
>
> Master/Slave Set: ms-opennms-data
> Masters: [ monitoring-node-01 ]
> Slaves: [ monitoring-node-02 ]
> Master/Slave Set: ms-opennms-db
> Masters: [ monitoring-node-01 ]
> Slaves: [ monitoring-node-02 ]
> Master/Slave Set: ms-opennms-config
> Masters: [ monitoring-node-01 ]
> Slaves: [ monitoring-node-02 ]
> Resource Group: dependencies
> vip (ocf::heartbeat:IPaddr2): Started monitoring-node-02
> fs-opennms-config (ocf::heartbeat:Filesystem): Stopped
> fs-opennms-db (ocf::heartbeat:Filesystem): Stopped
> fs-opennms-data (ocf::heartbeat:Filesystem): Stopped
> postgres (lsb:postgresql-8.4): Stopped
> opennms (lsb:opennms): Stopped
>
> Failed actions:
> fs-opennms-config_start_0 (node=monitoring-node-02, call=18, rc=1,
> status=complete): unknown error fs-opennms-config_start_0 (node=m)
>
>
> When I then type the following command:
>
> crm resource cleanup fs-opennms-config
>
> then the failover goes through and services are started on node01
>
>
> crm_mon
>
>
> ============
> Last updated: Mon Jul 5 13:31:31 2010
> Stack: openais
> Current DC: monitoring-node-01 - partition with quorum
> Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
> 2 Nodes configured, 2 expected votes
> 4 Resources configured.
> ============
>
> Online: [ monitoring-node-01 monitoring-node-02 ]
>
> Master/Slave Set: ms-opennms-data
> Masters: [ monitoring-node-01 ]
> Slaves: [ monitoring-node-02 ]
> Master/Slave Set: ms-opennms-db
> Masters: [ monitoring-node-01 ]
> Slaves: [ monitoring-node-02 ]
> Master/Slave Set: ms-opennms-config
> Masters: [ monitoring-node-01 ]
> Slaves: [ monitoring-node-02 ]
> Resource Group: dependencies
> vip (ocf::heartbeat:IPaddr2): Started monitoring-node-01
> fs-opennms-config (ocf::heartbeat:Filesystem): Started
> monitoring-node-01
> fs-opennms-db (ocf::heartbeat:Filesystem): Started
> monitoring-node-01
> fs-opennms-data (ocf::heartbeat:Filesystem): Started
> monitoring-node-01
> postgres (lsb:postgresql-8.4): Started monitoring-node-01
> opennms (lsb:opennms): Started monitoring-node-01
>
>
> There are some entries in my daemon.log[2] which look
> if they have something to do with my problem...
>
> monitoring-node-01 lrmd: [994]: info: rsc:drbd-opennms-data:1:30: promote
> monitoring-node-01 crmd: [998]: info: do_lrm_rsc_op: Performing
> key=93:25:0:d49b62be-1e33-48ca-a8c3-cb128676d444
> op=fs-opennms-config_start_0 )
> monitoring-node-01 lrmd: [994]: info: rsc:fs-opennms-config:31: start
> monitoring-node-01 Filesystem[2464]: INFO: Running start for
> /dev/drbd/by-res/config on /etc/opennms
> monitoring-node-01 lrmd: [994]: info: RA output:
> (fs-opennms-config:start:stderr) FATAL: Module scsi_hostadapter not found.
> monitoring-node-01 lrmd: [994]: info: RA output:
> (drbd-opennms-data:1:promote:stdout)
> monitoring-node-01 crmd: [998]: info: process_lrm_event: LRM operation
> drbd-opennms-data:1_promote_0 (call=30, rc=0, cib-update=35,
> confirmed=true) ok
> monitoring-node-01 lrmd: [994]: info: RA output:
> (fs-opennms-config:start:stderr) /dev/drbd/by-res/config: Wrong medium
> type
> monitoring-node-01 lrmd: [994]: info: RA output:
> (fs-opennms-config:start:stderr) mount: block device /dev/drbd0 is
> write-protected, mounting read-only
> monitoring-node-01 lrmd: [994]: info: RA output:
> (fs-opennms-config:start:stderr) mount: Wrong medium type
> monitoring-node-01 Filesystem[2464]: ERROR: Couldn't mount filesystem
> /dev/drbd/by-res/config on /etc/opennms
> monitoring-node-01 crmd: [998]: info: process_lrm_event: LRM operation
> fs-opennms-config_start_0 (call=31, rc=1, cib-update=36, confirmed=true)
> unknown error
>
> ...but I don't know how to troubleshoot it.
>
>
>
> [1] http://www.corosync.org/doku.php?id=faq:cisco_switches
> [2] http://pastebin.com/DKLjXtx8
>
>
Hi,
First you might want to look at the following error, see if the module
is available on both servers.
(fs-opennms-config:start:stderr) FATAL: Module scsi_hostadapter not found.
Then try to run the resource manually:
- go to /usr/lib/ocf/resource.d/heartbeat
- export OCF_ROOT=/usr/lib/ocf
- export OCF_RESKEY_device="/dev/drbd/by-res/config"
- export OCF_RESKEY_options=rw
- export OCF_RESKEY_fstype=xfs
- export OCF_RESKEY_directory="/etc/opennms"
- ./Filesystem start
See if you encounter any errors here. Run the steps on both servers.
Make sure to move the drbd resource from server to server so that the
mount works. You do that via
- go to server where drbd device is currently mounted and in a primary state
- umount /etc/opennms
- drbdadm secondary config
- move to other server
- drbdadm primary config
Also, make sure that pacemaker doesn't interfere with these operations :)
Cheers.
--
Dan FRINCU
Systems Engineer
CCNA, RHCE
More information about the Pacemaker
mailing list