[Pacemaker] [DRBD-user] DRBD active/passive on Pacemaker+CMAN cluster unexpectedly performs STONITH when promoting
Lars Ellenberg
lars.ellenberg at linbit.com
Fri Jul 4 12:40:33 UTC 2014
On Thu, Jul 03, 2014 at 04:05:36AM +0200, Giuseppe Ragusa wrote:
> Hi all,
> I deployed a 2 nodes (physical) RHCS Pacemaker cluster on CentOS 6.5 x86_64 (fully up-to-date) with:
>
> cman-3.0.12.1-59.el6_5.2.x86_64
> pacemaker-1.1.10-14.el6_5.3.x86_64
> pcs-0.9.90-2.el6.centos.3.noarch
> qemu-kvm-0.12.1.2-2.415.el6_5.10.x86_64
> qemu-kvm-tools-0.12.1.2-2.415.el6_5.10.x86_64
> drbd-utils-8.9.0-1.el6.x86_64
> drbd-udev-8.9.0-1.el6.x86_64
> drbd-rgmanager-8.9.0-1.el6.x86_64
> drbd-bash-completion-8.9.0-1.el6.x86_64
> drbd-pacemaker-8.9.0-1.el6.x86_64
> drbd-8.9.0-1.el6.x86_64
> drbd-km-2.6.32_431.20.3.el6.x86_64-8.4.5-1.x86_64
> kernel-2.6.32-431.20.3.el6.x86_64
>
> The aim is to run KVM virtual machines backed by DRBD (8.4.5) in an
> active/passive mode (no dual primary and so no live migration).
>
> Just to err on the side of consistency against HA (and to pave the way
> for a possible dual-primary live-migration-capable setup), I
> configured DRBD for resource-and-stonith with rhcs_fence (that's why I
> installed drbd-rgmanager) as fence-peer handler and stonith devices
> configured in Pacemaker (pcmk-redirect in cluster.conf).
>
> The setup "almost" works (all seems ok with: "pcs status", "crm_mon
> -Arf1", "corosync-cfgtool -s", "corosync-objctl | grep member") , but
> every time it needs a resource promotion (to Master, i.e. becoming
> primary) it either fails or fences the other node (the one supposed to
> become Slave i.e. secondary) and only then succeeds.
>
> It happens, for example both on initial resource definition (when
> attempting first start) and on node entering standby (when trying to
> automatically move the resources by stopping then starting them).
>
> I collected a full "pcs cluster report" and I can provide a CIB dump,
> but I will initially paste here an excerpt from my configuration just
> in case it happens to be a simple configuration error that someone can
> spot on the fly ;> (hoping...)
>
> Keep in mind that the setup has separated redundant network
> connections for LAN (1 Gib/s LACP to switches), Corosync (1 Gib/s
> roundrobin back-to-back) and DRBD (10 Gib/s roundrobin back-to-back)
> and that FQDNs are correctly resolved through /etc/hosts
Make sure youre DRBD are "Connected UpToDate/UpToDate"
before you let the cluster take over control of who is master.
> DRBD:
>
> /etc/drbd.d/global_common.conf:
>
> ------------------------------------------------------------------------------------------------------
>
> global {
> usage-count no;
> }
>
> common {
> protocol C;
> disk {
> on-io-error detach;
> fencing resource-and-stonith;
> disk-barrier no;
> disk-flushes no;
> al-extents 3389;
> c-plan-ahead 200;
> c-fill-target 15M;
> c-max-rate 100M;
> c-min-rate 10M;
> }
> net {
> after-sb-0pri discard-zero-changes;
> after-sb-1pri discard-secondary;
> after-sb-2pri disconnect;
> csums-alg sha1;
> data-integrity-alg sha1;
> max-buffers 8000;
> max-epoch-size 8000;
> unplug-watermark 16;
> sndbuf-size 0;
> verify-alg sha1;
> }
> startup {
> wfc-timeout 300;
> outdated-wfc-timeout 80;
> degr-wfc-timeout 120;
> }
> handlers {
> fence-peer "/usr/lib/drbd/rhcs_fence";
> }
> }
>
> ------------------------------------------------------------------------------------------------------
>
> Sample DRBD resource (there are others, similar)
> /etc/drbd.d/dc_vm.res:
>
> ------------------------------------------------------------------------------------------------------
>
> resource dc_vm {
> device /dev/drbd1;
> disk /dev/VolGroup00/dc_vm;
> meta-disk internal;
> on cluster1.verolengo.privatelan {
> address ipv4 172.16.200.1:7790;
> }
> on cluster2.verolengo.privatelan {
> address ipv4 172.16.200.2:7790;
> }
> }
>
> ------------------------------------------------------------------------------------------------------
>
> RHCS:
>
> /etc/cluster/cluster.conf
>
> ------------------------------------------------------------------------------------------------------
>
> <?xml version="1.0"?>
> <cluster name="vclu" config_version="14">
> <cman two_node="1" expected_votes="1" keyfile="/etc/corosync/authkey" transport="udpu" port="5405"/>
> <totem consensus="60000" join="6000" token="100000" token_retransmits_before_loss_const="20" rrp_mode="passive" secauth="on"/>
> <clusternodes>
> <clusternode name="cluster1.verolengo.privatelan" votes="1" nodeid="1">
> <altname name="clusterlan1.verolengo.privatelan" port="6405"/>
> <fence>
> <method name="pcmk-redirect">
> <device name="pcmk" port="cluster1.verolengo.privatelan"/>
> </method>
> </fence>
> </clusternode>
> <clusternode name="cluster2.verolengo.privatelan" votes="1" nodeid="2">
> <altname name="clusterlan2.verolengo.privatelan" port="6405"/>
> <fence>
> <method name="pcmk-redirect">
> <device name="pcmk" port="cluster2.verolengo.privatelan"/>
> </method>
> </fence>
> </clusternode>
> </clusternodes>
> <fencedevices>
> <fencedevice name="pcmk" agent="fence_pcmk"/>
> </fencedevices>
> <fence_daemon clean_start="0" post_fail_delay="30" post_join_delay="30"/>
> <logging debug="on"/>
> <rm disabled="1">
> <failoverdomains/>
> <resources/>
> </rm>
> </cluster>
>
> ------------------------------------------------------------------------------------------------------
>
> Pacemaker:
>
> PROPERTIES:
>
> pcs property set default-resource-stickiness=100
> pcs property set no-quorum-policy=ignore
>
> STONITH:
>
> pcs stonith create ilocluster1 fence_ilo2 action="off" delay="10" \
> ipaddr="ilocluster1.verolengo.privatelan" login="cluster2" passwd="test" power_wait="4" \
> pcmk_host_check="static-list" pcmk_host_list="cluster1.verolengo.privatelan" op monitor interval=60s
> pcs stonith create ilocluster2 fence_ilo2 action="off" \
> ipaddr="ilocluster2.verolengo.privatelan" login="cluster1" passwd="test" power_wait="4" \
> pcmk_host_check="static-list" pcmk_host_list="cluster2.verolengo.privatelan" op monitor interval=60s
> pcs stonith create pdu1 fence_apc action="off" \
> ipaddr="pdu1.verolengo.privatelan" login="cluster" passwd="test" \
> pcmk_host_map="cluster1.verolengo.privatelan:3,cluster1.verolengo.privatelan:4,cluster2.verolengo.privatelan:6,cluster2.verolengo.privatelan:7" \
> pcmk_host_check="static-list" pcmk_host_list="cluster1.verolengo.privatelan,cluster2.verolengo.privatelan" op monitor interval=60s
>
> pcs stonith level add 1 cluster1.verolengo.privatelan ilocluster1
> pcs stonith level add 2 cluster1.verolengo.privatelan pdu1
> pcs stonith level add 1 cluster2.verolengo.privatelan ilocluster2
> pcs stonith level add 2 cluster2.verolengo.privatelan pdu1
>
> pcs property set stonith-enabled=true
> pcs property set stonith-action=off
>
> SAMPLE RESOURCE:
>
> pcs cluster cib dc_cfg
> pcs -f dc_cfg resource create DCVMDisk ocf:linbit:drbd \
> drbd_resource=dc_vm op monitor interval="31s" role="Master" \
> op monitor interval="29s" role="Slave" \
> op start interval="0" timeout="120s" \
> op stop interval="0" timeout="180s"
> pcs -f dc_cfg resource master DCVMDiskClone DCVMDisk \
> master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 \
> notify=true target-role=Started is-managed=true
> pcs -f dc_cfg resource create DCVM ocf:heartbeat:VirtualDomain \
> config=/etc/libvirt/qemu/dc.xml migration_transport=tcp migration_network_suffix=-10g \
> hypervisor=qemu:///system meta allow-migrate=false target-role=Started is-managed=true \
> op start interval="0" timeout="120s" \
> op stop interval="0" timeout="120s" \
> op monitor interval="60s" timeout="120s"
> pcs -f dc_cfg constraint colocation add DCVM DCVMDiskClone INFINITY with-rsc-role=Master
> pcs -f dc_cfg constraint order promote DCVMDiskClone then start DCVM
> pcs -f dc_cfg constraint location DCVM prefers cluster2.verolengo.privatelan=50
> pcs cluster cib-push firewall_cfg
>
> Since I know that pcs still has some rough edges, I installed crmsh too, but never actually used it.
>
> Many thanks in advance for your attention.
>
> Kind regards,
> Giuseppe Ragusa
>
>
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed
More information about the Pacemaker
mailing list