[Pacemaker] DRBD Master/Slave in a 3 node cluster

Mon Sep 30 21:25:00 UTC 2013

Sorry maybe this is not the problem, but for avoid the split-brain you need
to configure fencing(stonith).

2013/9/30 James Oakley <jfunk at funktronics.ca>

> I am having some trouble with DRBD Master/Slave resources in a 3-node
> cluster.
>
> I am using the Pacemaker packages from ha-clustering:Stable on openSUSE
> 12.3. I was going to try the packages from Unstable to see if they work
> better, but it seems the openais package is missing there.
>
> So I have 3 nodes, called arthur, jonas, and rusty. The jonas and rusty
> nodes have 4 DRBD master/slave resources, which are used to back a
> series of filesystems, while the arthur node is included mainly to avoid
> split-brain, but I intend to run some resources on it as well, and
> possibly add some more nodes.
>
> I originally set up the DRBD resources manually, which was working fine,
> and I added the configurations to Pacemaker later, having disabled the
> drbd start script on both DRBD nodes.
>
> I set up location restraints to keep the resources only on jonas/rusty,
> with a -inf restraint for arthur.
>
> However, I can't seem to get it to run one of the nodes as a slave at
> all. I can get either of the nodes up as a master, but it seems to never
> set up the slave. I can manually bring up drbd on the secondary node
> using drbdadm, but if I run a reprobe/refresh, it will promptly shut
> them down, leaving only the masters.
>
> Failover obviously induces split-brain in DRBD, and if a node goes down,
> I have to manually massage the whole thing to get it working again. It's
> actually a LOT more work than if I just ran drbd and started everything
> manually without pacemaker.
>
> In crm status, it looks like this:
>
>  Master/Slave Set: ms_drbd_1 [res_drbd_1]
>      Masters: [ jonas.funktronics.ca ]
>      Stopped: [ res_drbd_1:1 ]
>
> If I restart openais or refresh, I get no obvious errors in the logs,
> just the drbd shutdown messages:
>
> 2013-09-29T23:08:54.053298-07:**00 rusty attrd[12394]:   notice:
> attrd_local_callback: Sending full refresh (origin=crmd)
> 2013-09-29T23:08:54.137456-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_IPv6addr_ipv6.venture_**monitor_0
> (call=9, rc=0, cib-update=9, confirmed=true) ok
> 2013-09-29T23:08:54.141872-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_rpcbind_rpcbind_monitor_0 (call=18,
> rc=0, cib-update=10, confirmed=true) ok
> 2013-09-29T23:08:54.148540-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_nfsserver_nfsserver_**monitor_0
> (call=13, rc=0, cib-update=11, confirmed=true) ok
> 2013-09-29T23:08:54.158315-07:**00 rusty exportfs(res_exportfs_root)[**13105]:
> INFO: Directory /srv/nfs is not exported to 10.7.41.0/24 (stopped).
> 2013-09-29T23:08:54.171138-07:**00 rusty exportfs(res_exportfs_backup)[**13127]:
> INFO: Directory /srv/nfs/backup is not exported to 10.7.41.0/24 (stopped).
> 2013-09-29T23:08:54.179063-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_exportfs_root_monitor_0 (call=23,
> rc=7, cib-update=12, confirmed=true) not running
> 2013-09-29T23:08:54.193543-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_IPaddr2_ip.venture_**monitor_0
> (call=5, rc=0, cib-update=13, confirmed=true) ok
> 2013-09-29T23:08:54.194124-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_exportfs_backup_monitor_0 (call=28,
> rc=7, cib-update=14, confirmed=true) not running
> 2013-09-29T23:08:54.241585-07:**00 rusty exportfs(res_exportfs_media)[**13162]:
> INFO: Directory /srv/nfs/media is not exported to 10.7.41.0/24 (stopped).
> 2013-09-29T23:08:54.241761-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_exportfs_media_monitor_0 (call=43,
> rc=7, cib-update=15, confirmed=true) not running
> 2013-09-29T23:08:54.242215-07:**00 rusty exportfs(res_exportfs_arthur)[**13196]:
> INFO: Directory /srv/nfs/arthur is not exported to arthur.funktronics.ca(stopped).
> 2013-09-29T23:08:54.242373-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_exportfs_arthur_monitor_0 (call=47,
> rc=7, cib-update=16, confirmed=true) not running
> 2013-09-29T23:08:54.249392-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_Xen_hank_monitor_0 (call=35, rc=7,
> cib-update=17, confirmed=true) not running
> 2013-09-29T23:08:54.250156-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_Xen_sproqhub_monitor_0 (call=39, rc=7,
> cib-update=18, confirmed=true) not running
> 2013-09-29T23:08:54.296841-07:**00 rusty cib[12391]:   notice:
> crm_ipc_prepare: Message exceeds the configured ipc limit (51200 bytes),
> consider configuring PCMK_ipc_buffer to 106150 or higher to avoid
> compression overheads
> 2013-09-29T23:08:54.395206-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_atftpd_atftpd_monitor_0 (call=59,
> rc=0, cib-update=19, confirmed=true) ok
> 2013-09-29T23:08:54.418934-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_IPaddr2_storage_monitor_0 (call=55,
> rc=0, cib-update=20, confirmed=true) ok
> 2013-09-29T23:08:54.508049-07:**00 rusty attrd[12394]:   notice:
> attrd_trigger_update: Sending flush op to all hosts for: master-res_drbd_1
> (10000)
> 2013-09-29T23:08:54.509311-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_Filesystem_backup_monitor_**0
> (call=70, rc=7, cib-update=21, confirmed=true) not running
> 2013-09-29T23:08:54.519015-07:**00 rusty attrd[12394]:   notice:
> attrd_perform_update: Sent update 4: master-res_drbd_1=10000
> 2013-09-29T23:08:54.521105-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_drbd_1_monitor_0 (call=65, rc=0,
> cib-update=22, confirmed=true) ok
> 2013-09-29T23:08:54.521116-07:**00 rusty attrd[12394]:   notice:
> attrd_trigger_update: Sending flush op to all hosts for: master-res_drbd_2
> (10000)
> 2013-09-29T23:08:54.521121-07:**00 rusty attrd[12394]:   notice:
> attrd_perform_update: Sent update 7: master-res_drbd_2=10000
> 2013-09-29T23:08:54.521124-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_Filesystem_media_monitor_0 (call=79,
> rc=7, cib-update=23, confirmed=true) not running
> 2013-09-29T23:08:54.521128-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_drbd_2_monitor_0 (call=75, rc=0,
> cib-update=24, confirmed=true) ok
> 2013-09-29T23:08:54.560687-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_Filesystem_breadmaster_**monitor_0
> (call=92, rc=7, cib-update=25, confirmed=true) not running
> 2013-09-29T23:08:54.580991-07:**00 rusty attrd[12394]:   notice:
> attrd_trigger_update: Sending flush op to all hosts for: master-res_drbd_3
> (10000)
> 2013-09-29T23:08:54.582406-07:**00 rusty attrd[12394]:   notice:
> attrd_perform_update: Sent update 12: master-res_drbd_3=10000
> 2013-09-29T23:08:54.582795-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_drbd_3_monitor_0 (call=84, rc=0,
> cib-update=26, confirmed=true) ok
> 2013-09-29T23:08:54.601145-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_Filesystem_arthur_monitor_**0
> (call=101, rc=7, cib-update=27, confirmed=true) not running
> 2013-09-29T23:08:54.609509-07:**00 rusty attrd[12394]:   notice:
> attrd_trigger_update: Sending flush op to all hosts for: master-res_drbd_4
> (10000)
> 2013-09-29T23:08:54.610446-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_drbd_4_monitor_0 (call=97, rc=0,
> cib-update=28, confirmed=true) ok
> 2013-09-29T23:08:54.704624-07:**00 rusty attrd[12394]:   notice:
> attrd_perform_update: Sent update 17: master-res_drbd_4=10000
> 2013-09-29T23:08:54.726503-07:**00 rusty attrd[12394]:   notice:
> attrd_perform_update: Sent update 22: master-res_drbd_4=10000
> 2013-09-29T23:08:59.236465-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_lxc_breadmaster_monitor_0 (call=109,
> rc=7, cib-update=29, confirmed=true) not running
> 2013-09-29T23:08:59.237788-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_drbd_1_notify_0 (call=111, rc=0,
> cib-update=0, confirmed=true) ok
> 2013-09-29T23:08:59.270440-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_drbd_3_notify_0 (call=117, rc=0,
> cib-update=0, confirmed=true) ok
> 2013-09-29T23:08:59.271697-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_drbd_2_notify_0 (call=115, rc=0,
> cib-update=0, confirmed=true) ok
> 2013-09-29T23:08:59.275311-07:**00 rusty attrd[12394]:   notice:
> attrd_trigger_update: Sending flush op to all hosts for: probe_complete
> (true)
> 2013-09-29T23:08:59.278171-07:**00 rusty crmd[12396]:   notice:
> process_lrm_event: LRM operation res_drbd_4_notify_0 (call=119, rc=0,
> cib-update=0, confirmed=true) ok
> 2013-09-29T23:08:59.332869-07:**00 rusty kernel: [ 1392.219061] block
> drbd1: peer( Secondary -> Unknown ) conn( Connected -> Disconnecting )
> pdsk( UpToDate -> DUnknown )
> 2013-09-29T23:08:59.333943-07:**00 rusty systemd[1]: Stopping LSB: launch
> atftpd server...
> 2013-09-29T23:08:59.338861-07:**00 rusty kernel: [ 1392.220259] block
> drbd1: asender terminated
> 2013-09-29T23:08:59.339364-07:**00 rusty kernel: [ 1392.220266] block
> drbd1: Terminating drbd1_asender
> 2013-09-29T23:08:59.339366-07:**00 rusty kernel: [ 1392.220281] block
> drbd3: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk(
> UpToDate -> DUnknown )
> 2013-09-29T23:08:59.339367-07:**00 rusty kernel: [ 1392.220330] block
> drbd2: peer( Secondary -> Unknown ) conn( Connected -> Disconnecting )
> pdsk( UpToDate -> DUnknown )
>
>
> Here is the config. I couldn't find any way to specify the master/slave
> hosts in the config, so I figured it just grabs them from the drbd
> resources. However, I have seen the cluster attempt to run drbd
> resources on arthur, which is odd. In fact, it occasionally attempts to
> run other services, such as the NFS resources, that have a -inf location
> constraint for arthur on arthur.
>
> node arthur.funktronics.ca \
>         attributes standby="off"
> node jonas.funktronics.ca \
>         attributes standby="off"
> node rusty.funktronics.ca
> primitive res_Filesystem_arthur ocf:heartbeat:Filesystem \
>         params device="/dev/drbd/by-res/**arthur"
> directory="/srv/nfs/arthur" fstype="ext4" \
>         operations $id="res_Filesystem_arthur-**operations" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40" start-delay="0" \
>         op notify interval="0" timeout="60"
> primitive res_Filesystem_backup ocf:heartbeat:Filesystem \
>         params device="/dev/drbd/by-res/**backup"
> directory="/srv/nfs/backup" fstype="ext4" \
>         operations $id="res_Filesystem_backup-**operations" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40" start-delay="0" \
>         op notify interval="0" timeout="60"
> primitive res_Filesystem_breadmaster ocf:heartbeat:Filesystem \
>         params device="/dev/drbd/by-res/**breadmaster"
> directory="/var/lib/lxc/**breadmaster/" fstype="ext4" \
>         operations $id="res_Filesystem_**breadmaster-operations" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40" start-delay="0" \
>         op notify interval="0" timeout="60" \
>         meta target-role="started"
> primitive res_Filesystem_media ocf:heartbeat:Filesystem \
>         params device="/dev/drbd/by-res/**media"
> directory="/srv/nfs/media" fstype="ext4" \
>         operations $id="res_Filesystem_media-**operations" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40" start-delay="0" \
>         op notify interval="0" timeout="60" \
>         meta target-role="started"
> primitive res_IPaddr2_ip.venture ocf:heartbeat:IPaddr2 \
>         params ip="10.7.41.24" nic="br0" cidr_netmask="24" \
>         operations $id="res_IPaddr2_ip.venture-**operations" \
>         op start interval="0" timeout="20" \
>         op stop interval="0" timeout="20" \
>         op monitor interval="10" timeout="20" start-delay="0" \
>         meta is-managed="true" target-role="started"
> primitive res_IPaddr2_storage ocf:heartbeat:IPaddr2 \
>         params ip="10.7.42.5" nic="br1" cidr_netmask="24" \
>         operations $id="res_IPaddr2_storage-**operations" \
>         op start interval="0" timeout="20" \
>         op stop interval="0" timeout="20" \
>         op monitor interval="10" timeout="20" start-delay="0"
> primitive res_IPv6addr_ipv6.venture ocf:heartbeat:IPv6addr \
>         params ipv6addr="2001:xxxxxx/64" \
>         operations $id="res_IPv6addr_ipv6.**venture-operations" \
>         op start interval="0" timeout="15" \
>         op stop interval="0" timeout="15" \
>         op monitor interval="15" timeout="15" start-delay="0" \
>         meta target-role="Started"
> primitive res_Xen_hank ocf:heartbeat:Xen \
>         params xmfile="/etc/xen/vm/hank" shutdown_acpi="false"
> allow_mem_management="false" \
>         operations $id="res_Xen_hank-operations" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="40" \
>         op monitor interval="10" timeout="30" start-delay="0" \
>         op migrate_from interval="0" timeout="120" \
>         op migrate_to interval="0" timeout="120" \
>         meta target-role="started"
> primitive res_Xen_sproqhub ocf:heartbeat:Xen \
>         params xmfile="/etc/xen/vm/sproqhub" shutdown_acpi="false"
> allow_mem_management="false" \
>         operations $id="res_Xen_sproqhub-**operations" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="40" \
>         op monitor interval="10" timeout="30" start-delay="0" \
>         op migrate_from interval="0" timeout="120" \
>         op migrate_to interval="0" timeout="120" \
>         meta target-role="started"
> primitive res_atftpd_atftpd systemd:atftpd \
>         operations $id="res_atftpd_atftpd-**operations" \
>         op start interval="0" timeout="15" \
>         op stop interval="0" timeout="15" \
>         op monitor interval="15" timeout="15" start-delay="15" \
>         meta target-role="Started"
> primitive res_drbd_1 ocf:linbit:drbd \
>         params drbd_resource="backup" \
>         operations $id="res_drbd_1-operations" \
>         op start interval="0" timeout="240" \
>         op promote interval="0" timeout="90" \
>         op demote interval="0" timeout="90" \
>         op stop interval="0" timeout="100" \
>         op monitor interval="10" timeout="20" start-delay="0" \
>         op notify interval="0" timeout="90" \
>         meta target-role="started"
> primitive res_drbd_2 ocf:linbit:drbd \
>         params drbd_resource="media" \
>         operations $id="res_drbd_2-operations" \
>         op start interval="0" timeout="240" \
>         op promote interval="0" timeout="90" \
>         op demote interval="0" timeout="90" \
>         op stop interval="0" timeout="100" \
>         op monitor interval="10" timeout="20" start-delay="0" \
>         op notify interval="0" timeout="90" \
>         meta target-role="started"
> primitive res_drbd_3 ocf:linbit:drbd \
>         params drbd_resource="breadmaster" \
>         operations $id="res_drbd_3-operations" \
>         op start interval="0" timeout="240" \
>         op promote interval="0" timeout="90" \
>         op demote interval="0" timeout="90" \
>         op stop interval="0" timeout="100" \
>         op monitor interval="10" timeout="20" start-delay="0" \
>         op notify interval="0" timeout="90" \
>         meta target-role="started"
> primitive res_drbd_4 ocf:linbit:drbd \
>         params drbd_resource="arthur" \
>         operations $id="res_drbd_4-operations" \
>         op start interval="0" timeout="240" \
>         op promote interval="0" timeout="90" \
>         op demote interval="0" timeout="90" \
>         op stop interval="0" timeout="100" \
>         op monitor interval="10" timeout="20" start-delay="0" \
>         op notify interval="0" timeout="90" \
>         meta target-role="started"
> primitive res_exportfs_arthur ocf:heartbeat:exportfs \
>         params clientspec="arthur.**funktronics.ca<http://arthur.funktronics.ca>"
> options="rw,no_root_squash" directory="/srv/nfs/arthur" fsid="4"
> unlock_on_stop="false" wait_for_leasetime_on_stop="**false" \
>         operations $id="res_exportfs_arthur-**operations" \
>         op start interval="0" timeout="40" \
>         op stop interval="0" timeout="10" \
>         op monitor interval="10" timeout="20" start-delay="0" \
>         meta target-role="started" is-managed="true"
> primitive res_exportfs_backup ocf:heartbeat:exportfs \
>         params clientspec="10.7.41.0/24" options="rw,no_root_squash"
> directory="/srv/nfs/backup" fsid="1" unlock_on_stop="true"
> wait_for_leasetime_on_stop="**false" \
>         operations $id="res_exportfs_backup-**operations" \
>         op start interval="0" timeout="40" \
>         op stop interval="0" timeout="10" \
>         op monitor interval="10" timeout="20" start-delay="0" \
>         meta is-managed="true"
> primitive res_exportfs_media ocf:heartbeat:exportfs \
>         params clientspec="10.7.41.0/24" options="rw,no_root_squash,**async"
> directory="/srv/nfs/media" fsid="2" unlock_on_stop="false"
> wait_for_leasetime_on_stop="**false" \
>         operations $id="res_exportfs_media-**operations" \
>         op start interval="0" timeout="40" \
>         op stop interval="0" timeout="10" \
>         op monitor interval="10" timeout="20" start-delay="0"
> primitive res_exportfs_root ocf:heartbeat:exportfs \
>         params clientspec="10.7.41.0/24" options="ro,crossmnt"
> directory="/srv/nfs" fsid="root" unlock_on_stop="true"
> wait_for_leasetime_on_stop="**false" \
>         operations $id="res_exportfs_root-**operations" \
>         op start interval="0" timeout="40" \
>         op stop interval="0" timeout="10" \
>         op monitor interval="10" timeout="20" start-delay="0" \
>         meta target-role="started"
> primitive res_lxc_breadmaster ocf:heartbeat:lxc \
>         params container="breadmaster" config="/var/lib/lxc/**breadmaster/config"
> \
>         operations $id="res_lxc_breadmaster-**operations" \
>         op start interval="0" timeout="10" \
>         op stop interval="0" timeout="30" \
>         op monitor interval="60" timeout="20" start-delay="0"
> primitive res_nfsserver_nfsserver lsb:nfsserver \
>         operations $id="res_nfsserver_nfsserver-**operations" \
>         op start interval="0" timeout="15" \
>         op stop interval="0" timeout="15" \
>         op monitor interval="15" timeout="15" start-delay="15"
> primitive res_rpcbind_rpcbind lsb:rpcbind \
>         operations $id="res_rpcbind_rpcbind-**operations" \
>         op start interval="0" timeout="15" \
>         op stop interval="0" timeout="15" \
>         op monitor interval="15" timeout="15" start-delay="15"
> ms ms_drbd_1 res_drbd_1 \
>         meta clone-max="2" notify="true" interleave="true"
> target-role="Master"
> ms ms_drbd_2 res_drbd_2 \
>         meta clone-max="2" notify="true" interleave="true"
> target-role="Master"
> ms ms_drbd_3 res_drbd_3 \
>         meta clone-max="2" notify="true" interleave="true"
> target-role="Master"
> ms ms_drbd_4 res_drbd_4 \
>         meta clone-max="2" notify="true" interleave="true"
> target-role="Master"
> location loc_ms_drbd_1_arthur.**funktronics.ca<http://loc_ms_drbd_1_arthur.funktronics.ca>ms_drbd_1 -inf:
> arthur.funktronics.ca
> location loc_ms_drbd_1_jonas.**funktronics.ca<http://loc_ms_drbd_1_jonas.funktronics.ca>ms_drbd_1 0:
> jonas.funktronics.ca
> location loc_ms_drbd_1_rusty.**funktronics.ca<http://loc_ms_drbd_1_rusty.funktronics.ca>ms_drbd_1 0:
> rusty.funktronics.ca
> location loc_ms_drbd_2_arthur.**funktronics.ca<http://loc_ms_drbd_2_arthur.funktronics.ca>ms_drbd_2 -inf:
> arthur.funktronics.ca
> location loc_ms_drbd_2_jonas.**funktronics.ca<http://loc_ms_drbd_2_jonas.funktronics.ca>ms_drbd_2 0:
> jonas.funktronics.ca
> location loc_ms_drbd_2_rusty.**funktronics.ca<http://loc_ms_drbd_2_rusty.funktronics.ca>ms_drbd_2 0:
> rusty.funktronics.ca
> location loc_ms_drbd_3_arthur.**funktronics.ca<http://loc_ms_drbd_3_arthur.funktronics.ca>ms_drbd_3 -inf:
> arthur.funktronics.ca
> location loc_ms_drbd_3_jonas.**funktronics.ca<http://loc_ms_drbd_3_jonas.funktronics.ca>ms_drbd_3 0:
> jonas.funktronics.ca
> location loc_ms_drbd_3_rusty.**funktronics.ca<http://loc_ms_drbd_3_rusty.funktronics.ca>ms_drbd_3 0:
> rusty.funktronics.ca
> location loc_ms_drbd_4_arthur.**funktronics.ca<http://loc_ms_drbd_4_arthur.funktronics.ca>ms_drbd_4 -inf:
> arthur.funktronics.ca
> location loc_ms_drbd_4_jonas.**funktronics.ca<http://loc_ms_drbd_4_jonas.funktronics.ca>ms_drbd_4 0:
> jonas.funktronics.ca
> location loc_ms_drbd_4_rusty.**funktronics.ca<http://loc_ms_drbd_4_rusty.funktronics.ca>ms_drbd_4 0:
> rusty.funktronics.ca
> location loc_res_Filesystem_arthur_**arthur.funktronics.ca<http://loc_res_Filesystem_arthur_arthur.funktronics.ca>res_Filesystem_arthur -inf:
> arthur.funktronics.ca
> location loc_res_Filesystem_arthur_**jonas.funktronics.ca<http://loc_res_Filesystem_arthur_jonas.funktronics.ca>res_Filesystem_arthur 0:
> jonas.funktronics.ca
> location loc_res_Filesystem_arthur_**rusty.funktronics.ca<http://loc_res_Filesystem_arthur_rusty.funktronics.ca>res_Filesystem_arthur 0:
> rusty.funktronics.ca
> location loc_res_Filesystem_backup_**arthur.funktronics.ca<http://loc_res_Filesystem_backup_arthur.funktronics.ca>res_Filesystem_backup -inf:
> arthur.funktronics.ca
> location loc_res_Filesystem_backup_**jonas.funktronics.ca<http://loc_res_Filesystem_backup_jonas.funktronics.ca>res_Filesystem_backup 0:
> jonas.funktronics.ca
> location loc_res_Filesystem_backup_**rusty.funktronics.ca<http://loc_res_Filesystem_backup_rusty.funktronics.ca>res_Filesystem_backup 0:
> rusty.funktronics.ca
> location loc_res_Filesystem_**breadmaster_arthur.**funktronics.ca<http://loc_res_Filesystem_breadmaster_arthur.funktronics.ca>res_Filesystem_breadmaster -inf:
> arthur.funktronics.ca
> location loc_res_Filesystem_**breadmaster_jonas.funktronics.**ca<http://loc_res_Filesystem_breadmaster_jonas.funktronics.ca>res_Filesystem_breadmaster 0:
> jonas.funktronics.ca
> location loc_res_Filesystem_**breadmaster_rusty.funktronics.**ca<http://loc_res_Filesystem_breadmaster_rusty.funktronics.ca>res_Filesystem_breadmaster 0:
> rusty.funktronics.ca
> location loc_res_Filesystem_media_**arthur.funktronics.ca<http://loc_res_Filesystem_media_arthur.funktronics.ca>res_Filesystem_media -inf:
> arthur.funktronics.ca
> location loc_res_Filesystem_media_**jonas.funktronics.ca<http://loc_res_Filesystem_media_jonas.funktronics.ca>res_Filesystem_media 0:
> jonas.funktronics.ca
> location loc_res_Filesystem_media_**rusty.funktronics.ca<http://loc_res_Filesystem_media_rusty.funktronics.ca>res_Filesystem_media 0:
> rusty.funktronics.ca
> location loc_res_IPaddr2_ip.venture_**arthur.funktronics.ca<http://loc_res_IPaddr2_ip.venture_arthur.funktronics.ca>res_IPaddr2_ip.venture -inf:
> arthur.funktronics.ca
> location loc_res_IPaddr2_ip.venture_**jonas.funktronics.ca<http://loc_res_IPaddr2_ip.venture_jonas.funktronics.ca>res_IPaddr2_ip.venture 2:
> jonas.funktronics.ca
> location loc_res_IPaddr2_ip.venture_**rusty.funktronics.ca<http://loc_res_IPaddr2_ip.venture_rusty.funktronics.ca>res_IPaddr2_ip.venture 0:
> rusty.funktronics.ca
> location loc_res_IPaddr2_storage_**arthur.funktronics.ca<http://loc_res_IPaddr2_storage_arthur.funktronics.ca>res_IPaddr2_storage -inf:
> arthur.funktronics.ca
> location loc_res_IPaddr2_storage_jonas.**funktronics.ca<http://loc_res_IPaddr2_storage_jonas.funktronics.ca>res_IPaddr2_storage 0:
> jonas.funktronics.ca
> location loc_res_IPaddr2_storage_rusty.**funktronics.ca<http://loc_res_IPaddr2_storage_rusty.funktronics.ca>res_IPaddr2_storage 0:
> rusty.funktronics.ca
> location loc_res_IPv6addr_ipv6.venture_**arthur.funktronics.ca<http://loc_res_IPv6addr_ipv6.venture_arthur.funktronics.ca>res_IPv6addr_ipv6.venture -inf:
> arthur.funktronics.ca
> location loc_res_IPv6addr_ipv6.venture_**jonas.funktronics.ca<http://loc_res_IPv6addr_ipv6.venture_jonas.funktronics.ca>res_IPv6addr_ipv6.venture 2:
> jonas.funktronics.ca
> location loc_res_IPv6addr_ipv6.venture_**rusty.funktronics.ca<http://loc_res_IPv6addr_ipv6.venture_rusty.funktronics.ca>res_IPv6addr_ipv6.venture 0:
> rusty.funktronics.ca
> location loc_res_Xen_hank_arthur.**funktronics.ca<http://loc_res_Xen_hank_arthur.funktronics.ca>res_Xen_hank -inf:
> arthur.funktronics.ca
> location loc_res_Xen_hank_jonas.**funktronics.ca<http://loc_res_Xen_hank_jonas.funktronics.ca>res_Xen_hank 0:
> jonas.funktronics.ca
> location loc_res_Xen_hank_rusty.**funktronics.ca<http://loc_res_Xen_hank_rusty.funktronics.ca>res_Xen_hank -inf:
> rusty.funktronics.ca
> location loc_res_Xen_sproqhub_arthur.**funktronics.ca<http://loc_res_Xen_sproqhub_arthur.funktronics.ca>res_Xen_sproqhub -inf:
> arthur.funktronics.ca
> location loc_res_Xen_sproqhub_jonas.**funktronics.ca<http://loc_res_Xen_sproqhub_jonas.funktronics.ca>res_Xen_sproqhub 0:
> jonas.funktronics.ca
> location loc_res_Xen_sproqhub_rusty.**funktronics.ca<http://loc_res_Xen_sproqhub_rusty.funktronics.ca>res_Xen_sproqhub -inf:
> rusty.funktronics.ca
> location loc_res_atftpd_atftpd_arthur.**funktronics.ca<http://loc_res_atftpd_atftpd_arthur.funktronics.ca>res_atftpd_atftpd -inf:
> arthur.funktronics.ca
> location loc_res_atftpd_atftpd_jonas.**funktronics.ca<http://loc_res_atftpd_atftpd_jonas.funktronics.ca>res_atftpd_atftpd 0:
> jonas.funktronics.ca
> location loc_res_atftpd_atftpd_rusty.**funktronics.ca<http://loc_res_atftpd_atftpd_rusty.funktronics.ca>res_atftpd_atftpd 0:
> rusty.funktronics.ca
> location loc_res_exportfs_arthur_**arthur.funktronics.ca<http://loc_res_exportfs_arthur_arthur.funktronics.ca>res_exportfs_arthur -inf:
> arthur.funktronics.ca
> location loc_res_exportfs_arthur_jonas.**funktronics.ca<http://loc_res_exportfs_arthur_jonas.funktronics.ca>res_exportfs_arthur 2:
> jonas.funktronics.ca
> location loc_res_exportfs_arthur_rusty.**funktronics.ca<http://loc_res_exportfs_arthur_rusty.funktronics.ca>res_exportfs_arthur 0:
> rusty.funktronics.ca
> location loc_res_exportfs_backup_**arthur.funktronics.ca<http://loc_res_exportfs_backup_arthur.funktronics.ca>res_exportfs_backup -inf:
> arthur.funktronics.ca
> location loc_res_exportfs_backup_jonas.**funktronics.ca<http://loc_res_exportfs_backup_jonas.funktronics.ca>res_exportfs_backup 0:
> jonas.funktronics.ca
> location loc_res_exportfs_backup_rusty.**funktronics.ca<http://loc_res_exportfs_backup_rusty.funktronics.ca>res_exportfs_backup 0:
> rusty.funktronics.ca
> location loc_res_exportfs_media_arthur.**funktronics.ca<http://loc_res_exportfs_media_arthur.funktronics.ca>res_exportfs_media -inf:
> arthur.funktronics.ca
> location loc_res_exportfs_media_jonas.**funktronics.ca<http://loc_res_exportfs_media_jonas.funktronics.ca>res_exportfs_media 2:
> jonas.funktronics.ca
> location loc_res_exportfs_media_rusty.**funktronics.ca<http://loc_res_exportfs_media_rusty.funktronics.ca>res_exportfs_media 0:
> rusty.funktronics.ca
> location loc_res_exportfs_root_arthur.**funktronics.ca<http://loc_res_exportfs_root_arthur.funktronics.ca>res_exportfs_root -inf:
> arthur.funktronics.ca
> location loc_res_exportfs_root_jonas.**funktronics.ca<http://loc_res_exportfs_root_jonas.funktronics.ca>res_exportfs_root 0:
> jonas.funktronics.ca
> location loc_res_exportfs_root_rusty.**funktronics.ca<http://loc_res_exportfs_root_rusty.funktronics.ca>res_exportfs_root 0:
> rusty.funktronics.ca
> location loc_res_lxc_breadmaster_**arthur.funktronics.ca<http://loc_res_lxc_breadmaster_arthur.funktronics.ca>res_lxc_breadmaster -inf:
> arthur.funktronics.ca
> location loc_res_lxc_breadmaster_jonas.**funktronics.ca<http://loc_res_lxc_breadmaster_jonas.funktronics.ca>res_lxc_breadmaster 0:
> jonas.funktronics.ca
> location loc_res_lxc_breadmaster_rusty.**funktronics.ca<http://loc_res_lxc_breadmaster_rusty.funktronics.ca>res_lxc_breadmaster 0:
> rusty.funktronics.ca
> location loc_res_nfsserver_nfsserver_**arthur.funktronics.ca<http://loc_res_nfsserver_nfsserver_arthur.funktronics.ca>res_nfsserver_nfsserver -inf:
> arthur.funktronics.ca
> location loc_res_nfsserver_nfsserver_**jonas.funktronics.ca<http://loc_res_nfsserver_nfsserver_jonas.funktronics.ca>res_nfsserver_nfsserver 0:
> jonas.funktronics.ca
> location loc_res_nfsserver_nfsserver_**rusty.funktronics.ca<http://loc_res_nfsserver_nfsserver_rusty.funktronics.ca>res_nfsserver_nfsserver 0:
> rusty.funktronics.ca
> location loc_res_rpcbind_rpcbind_**arthur.funktronics.ca<http://loc_res_rpcbind_rpcbind_arthur.funktronics.ca>res_rpcbind_rpcbind -inf:
> arthur.funktronics.ca
> location loc_res_rpcbind_rpcbind_jonas.**funktronics.ca<http://loc_res_rpcbind_rpcbind_jonas.funktronics.ca>res_rpcbind_rpcbind 0:
> jonas.funktronics.ca
> location loc_res_rpcbind_rpcbind_rusty.**funktronics.ca<http://loc_res_rpcbind_rpcbind_rusty.funktronics.ca>res_rpcbind_rpcbind 0:
> rusty.funktronics.ca
> colocation col_res_Filesystem_arthur_ms_**drbd_4 inf:
> res_Filesystem_arthur ms_drbd_4:Master
> colocation col_res_Filesystem_backup_ms_**drbd_1 inf:
> res_Filesystem_backup ms_drbd_1:Master
> colocation col_res_Filesystem_**breadmaster_ms_drbd_3 inf:
> res_Filesystem_breadmaster ms_drbd_3:Master
> colocation col_res_Filesystem_media_ms_**drbd_2 inf: res_Filesystem_media
> ms_drbd_2:Master
> colocation col_res_IPaddr2_storage_res_**IPaddr2_ip.venture inf:
> res_IPaddr2_storage res_IPaddr2_ip.venture
> colocation col_res_IPv6addr_ipv6.venture_**ms_drbd_1 inf: ms_drbd_1
> res_IPv6addr_ipv6.venture
> colocation col_res_IPv6addr_ipv6.venture_**ms_drbd_2 inf: ms_drbd_2
> res_IPv6addr_ipv6.venture
> colocation col_res_IPv6addr_ipv6.venture_**ms_drbd_3 inf: ms_drbd_3
> res_IPv6addr_ipv6.venture
> colocation col_res_IPv6addr_ipv6.venture_**ms_drbd_4 inf: ms_drbd_4
> res_IPv6addr_ipv6.venture
> colocation col_res_IPv6addr_ipv6.venture_**res_IPaddr2_ip.venture inf:
> res_IPv6addr_ipv6.venture res_IPaddr2_ip.venture
> colocation col_res_atftpd_atftpd_res_**IPaddr2_storage inf:
> res_atftpd_atftpd res_IPaddr2_storage
> colocation col_res_exportfs_arthur_res_**Filesystem_arthur inf:
> res_exportfs_arthur res_Filesystem_arthur
> colocation col_res_exportfs_backup_res_**Filesystem_backup inf:
> res_exportfs_backup res_Filesystem_backup
> colocation col_res_exportfs_media_res_**Filesystem_media inf:
> res_exportfs_media res_Filesystem_media
> colocation col_res_exportfs_root_res_**exportfs_arthur inf:
> res_exportfs_arthur res_exportfs_root
> colocation col_res_exportfs_root_res_**exportfs_backup inf:
> res_exportfs_backup res_exportfs_root
> colocation col_res_exportfs_root_res_**exportfs_media inf:
> res_exportfs_media res_exportfs_root
> colocation col_res_lxc_breadmaster_res_**Filesystem_breadmaster inf:
> res_lxc_breadmaster res_Filesystem_breadmaster
> colocation col_res_nfsserver_nfsserver_**res_exportfs_root inf:
> res_exportfs_root res_nfsserver_nfsserver
> colocation col_res_rpcbind_rpcbind_res_**IPv6addr_ipv6.venture inf:
> res_rpcbind_rpcbind res_IPv6addr_ipv6.venture
> colocation col_res_rpcbind_rpcbind_res_**nfsserver_nfsserver inf:
> res_nfsserver_nfsserver res_rpcbind_rpcbind
> order ord_ms_drbd_1_res_Filesystem_**backup inf: ms_drbd_1:promote
> res_Filesystem_backup:start
> order ord_ms_drbd_2_res_Filesystem_**media inf: ms_drbd_2:promote
> res_Filesystem_media:start
> order ord_ms_drbd_3_res_Filesystem_**breadmaster inf: ms_drbd_3:promote
> res_Filesystem_breadmaster:**start
> order ord_ms_drbd_4_res_Filesystem_**arthur inf: ms_drbd_4:promote
> res_Filesystem_arthur:start
> order ord_res_Filesystem_arthur_res_**exportfs_arthur inf:
> res_Filesystem_arthur res_exportfs_arthur
> order ord_res_Filesystem_backup_res_**exportfs_backup inf:
> res_Filesystem_backup res_exportfs_backup
> order ord_res_Filesystem_**breadmaster_res_lxc_**breadmaster inf:
> res_Filesystem_breadmaster res_lxc_breadmaster
> order ord_res_Filesystem_media_res_**exportfs_media inf:
> res_Filesystem_media res_exportfs_media
> order ord_res_IPaddr2_ip.venture_**res_IPaddr2_storage inf:
> res_IPaddr2_ip.venture res_IPaddr2_storage
> order ord_res_IPaddr2_ip.venture_**res_IPv6addr_ipv6.venture inf:
> res_IPaddr2_ip.venture res_IPv6addr_ipv6.venture
> order ord_res_IPaddr2_storage_res_**atftpd_atftpd inf:
> res_IPaddr2_storage res_atftpd_atftpd
> order ord_res_IPv6addr_ipv6.venture_**ms_drbd_1 inf:
> res_IPv6addr_ipv6.venture ms_drbd_1
> order ord_res_IPv6addr_ipv6.venture_**ms_drbd_2 inf:
> res_IPv6addr_ipv6.venture ms_drbd_2
> order ord_res_IPv6addr_ipv6.venture_**ms_drbd_3 inf:
> res_IPv6addr_ipv6.venture ms_drbd_3
> order ord_res_IPv6addr_ipv6.venture_**ms_drbd_4 inf:
> res_IPv6addr_ipv6.venture ms_drbd_4
> order ord_res_IPv6addr_ipv6.venture_**res_rpcbind_rpcbind inf:
> res_IPv6addr_ipv6.venture res_rpcbind_rpcbind
> order ord_res_exportfs_root_res_**exportfs_arthur inf: res_exportfs_root
> res_exportfs_arthur
> order ord_res_exportfs_root_res_**exportfs_backup inf: res_exportfs_root
> res_exportfs_backup
> order ord_res_exportfs_root_res_**exportfs_media inf: res_exportfs_root
> res_exportfs_media
> order ord_res_nfsserver_nfsserver_**res_exportfs_root inf:
> res_nfsserver_nfsserver res_exportfs_root
> order ord_res_rpcbind_rpcbind_res_**nfsserver_nfsserver inf:
> res_rpcbind_rpcbind res_nfsserver_nfsserver
> property $id="cib-bootstrap-options" \
>         expected-quorum-votes="3" \
>         stonith-enabled="false" \
>         symmetric-cluster="false" \
>         dc-version="1.1.9-2db99f1" \
>         no-quorum-policy="freeze" \
>         cluster-infrastructure="**classic openais (with plugin)" \
>         last-lrm-refresh="1380520394"
>
>
> Is there anything obvious I am missing?
>
> --
> James Oakley
> jfunk at funktronics.ca
>
> ______________________________**_________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/**mailman/listinfo/pacemaker<http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/**doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
> Bugs: http://bugs.clusterlabs.org
>

-- 
esta es mi vida e me la vivo hasta que dios quiera
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20130930/790dd6c8/attachment.htm>