[Pacemaker] Corosync quorum not updating on split node

Thu Sep 5 10:44:09 UTC 2013

Hi all,

I have a problem whereby when I create a network split/partition (by dropping traffic with iptables), the victim node for some reason does not realise it has split from the network.

It seems to recognise that it can't form a cluster due to network issues, but the status is not reflected in the output from corosync-quorumtool, and cluster services (via pacemaker) still continue to run. However, the other nodes in the rest of the cluster do realise they have lost contact with a node, no longer have quorum and correctly shut down services.

When I block traffic on the victim node's eth0, The remaining nodes see that they cannot communicate with it and shutdown :

# corosync-quorumtool -s
Version:          1.4.5
Nodes:            3
Ring ID:          696
Quorum type:      corosync_votequorum
Quorate:          No
Node votes:       1
Expected votes:   7
Highest expected: 7
Total votes:      3
Quorum:           4 Activity blocked
Flags:

However, the victim node still thinks everything is fine, and maintains a view of the cluster prior to the split :

# corosync-quorumtool -s
Version:          1.4.5
Nodes:            4
Ring ID:          716
Quorum type:      corosync_votequorum
Quorate:          Yes
Node votes:       1
Expected votes:   7
Highest expected: 7
Total votes:      4
Quorum:           4
Flags:            Quorate

However, it does notice in the logs that it cannot now form cluster, as the following messages repeat constantly :

corosync [MAIN  ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.

I would expect at this point for it to be in it's own network partition with a total of 1 vote, and block activity. However, this does not seem to happen until just after it rejoins the cluster. When I unblock traffic and it rejoins, I see the victim finally realise it had lost quorum :

Sep 05 09:52:21 corosync [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 720: memb=1, new=0, lost=3
Sep 05 09:52:21 corosync [VOTEQ ] quorum lost, blocking activity
Sep 05 09:52:21 corosync [QUORUM] This node is within the non-primary component and will NOT provide any services.
Sep 05 09:52:21 corosync [QUORUM] Members[1]: 358898186

And a second or so later it regains quorum :

crmd:   notice: ais_dispatch_message:         Membership 736: quorum acquired

So my question is why, when it realises it cannot form a cluster ("Totem in unable to form..."), does it not loose quorum, update the status as reflected by quorumtool and shutdown cluster services ?

Configuration file example and package versions/environment listed below. I'm using "updu" protocol as we need to avoid multicast in this environment; it will eventually be using a routed network. This behaviour also persists when I disable the pacemaker plugin and just test with corosync.

compatibility: whitetank
totem {
    version: 2
    secauth: off
    interface {
        member {
            memberaddr: 10.90.100.20
        }
        member {
            memberaddr: 10.90.100.21
        }
...
... more nodes snipped
...
ringnumber: 0
        bindnetaddr: 10.90.100.20
        mcastport: 5405
    }
    transport: udpu
}
amf {
        mode: disabled
}
aisexec {
        user: root
        group: root
}
quorum {
        provider: corosync_votequorum
        expected_votes: 7
}
service {
        # Load the Pacemaker Cluster Resource Manager
        name: pacemaker
        ver: 0
}

Environment : CentOS 6.4
Packages from OpenSUSE : http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/RedHat_RHEL-6/x86_64/
# rpm -qa | egrep "^(cluster|corosync|crm|libqb|pacemaker|resource-agents)" | sort
cluster-glue-1.0.11-3.1.x86_64
cluster-glue-libs-1.0.11-3.1.x86_64
corosync-1.4.5-2.2.x86_64
corosynclib-1.4.5-2.2.x86_64
crmsh-1.2.6-0.rc3.3.1.x86_64
libqb0-0.14.4-1.2.x86_64
pacemaker-1.1.9-2.1.x86_64
pacemaker-cli-1.1.9-2.1.x86_64
pacemaker-cluster-libs-1.1.9-2.1.x86_64
pacemaker-libs-1.1.9-2.1.x86_64
resource-agents-3.9.5-3.1.x86_64

Regards,

-Mark

________________________________

Mark Round
Senior Systems Administrator
NCC Group
Kings Court
Kingston Road
Leatherhead, KT22 7SL

Telephone: +44 1372 383815
Mobile: +44 7790 770413
Fax:
Website: www.nccgroup.com<http://www.nccgroup.com>
Email:  Mark.Round at nccgroup.com<mailto:Mark.Round at nccgroup.com>
        [http://www.nccgroup.com/media/192418/nccgrouplogo.jpg] <http://www.nccgroup.com/>
________________________________

This email is sent for and on behalf of NCC Group. NCC Group is the trading name of NCC Group Performance Testing Limited (Registered in England CRN: 4069379). Registered Office: Manchester Technology Centre, Oxford Road, Manchester, M1 7EF. The ultimate holding company is NCC Group plc (Registered in England CRN: 4627044).

Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than NCC Group or the sender of its existence.

For more information about NCC Group please visit www.nccgroup.com<http://www.nccgroup.com>

P Before you print think about the ENVIRONMENT

For more information please visit <a href="http://www.mimecast.com">http://www.mimecast.com<br>
This email message has been delivered safely and archived online by Mimecast.
</a>