[Pacemaker] Stopping heartbeat service on one node lead to restart of resources on other node in cluster

[Andrew Beekhof]

I'm reasonably sure this was a bug we fixed in 1.0.12
Could you update and retest?

On Tue, Feb 7, 2012 at 9:35 PM, neha chatrath <nehachatrath at gmail.com> wrote:
> Hello,
> I have a 2 node cluster with following configuration:
> *node $id="9e53a111-0dca-496c-9461-a38f3eec4d0e" mcg2 \
>        attributes standby="off"
> node $id="a90981f8-d993-4411-89f4-aff7156136d2" mcg1 \
>        attributes standby="off"
> primitive ClusterIP ocf:mcg:MCG_VIPaddr_RA \
>        params ip="192.168.115.50" cidr_netmask="255.255.255.0"
> nic="bond1.115:1" \
>        op monitor interval="40" timeout="20" \
>        meta target-role="Started"
> primitive EMS ocf:heartbeat:jboss \
>        params jboss_home="/opt/jboss-5.1.0.GA"
> java_home="/opt/jdk1.6.0_29/" \
>        op start interval="0" timeout="240" \
>        op stop interval="0" timeout="240" \
>        op monitor interval="30s" timeout="40s"
> primitive NDB_MGMT ocf:mcg:NDB_MGM_RA \
>        op monitor interval="120" timeout="120"
> primitive NDB_VIP ocf:heartbeat:IPaddr2 \
>        params ip="192.168.117.50" cidr_netmask="255.255.255.255"
> nic="bond0.117:1" \
>        op monitor interval="30" timeout="10"
> primitive Rmgr ocf:mcg:RM_RA \
>        op monitor interval="60" role="Master" timeout="30"
> on-fail="restart" \
>        op monitor interval="40" role="Slave" timeout="40" on-fail="restart"
> primitive Tmgr ocf:mcg:TM_RA \
>        op monitor interval="60" role="Master" timeout="30"
> on-fail="restart" \
>        op monitor interval="40" role="Slave" timeout="40" on-fail="restart"
> primitive mysql ocf:mcg:MYSQLD_RA \
>        op monitor interval="180" timeout="200"
> primitive ndbd ocf:mcg:NDBD_RA \
>        op monitor interval="120" timeout="120"
> primitive pimd ocf:mcg:PIMD_RA \
>        op monitor interval="60" role="Master" timeout="30"
> on-fail="restart" \
>        op monitor interval="40" role="Slave" timeout="40" on-fail="restart"
> ms ms_Rmgr Rmgr \
>        meta master-max="1" master-max-node="1" clone-max="2"
> clone-node-max="1" interleave="true" notify="true"
> ms ms_Tmgr Tmgr \
>        meta master-max="1" master-max-node="1" clone-max="2"
> clone-node-max="1" interleave="true" notify="true"
> ms ms_pimd pimd \
>        meta master-max="1" master-max-node="1" clone-max="2"
> clone-node-max="1" interleave="true" notify="true"
> clone EMS_CLONE EMS \
>        meta globally-unique="false" clone-max="2" clone-node-max="1"
> target-role="Started"
> clone mysqld_clone mysql \
>        meta globally-unique="false" clone-max="2" clone-node-max="1"
> clone ndbdclone ndbd \
>        meta globally-unique="false" clone-max="2" clone-node-max="1"
> target-role="Started"
> colocation ip_with_Pimd inf: ClusterIP ms_pimd:Master
> colocation ip_with_RM inf: ClusterIP ms_Rmgr:Master
> colocation ip_with_TM inf: ClusterIP ms_Tmgr:Master
> colocation ndb_vip-with-ndb_mgm inf: NDB_MGMT NDB_VIP
> order RM-after-mysqld inf: mysqld_clone ms_Rmgr
> order TM-after-RM inf: ms_Rmgr ms_Tmgr
> order ip-after-pimd inf: ms_pimd ClusterIP
> order mysqld-after-ndbd inf: ndbdclone mysqld_clone
> order pimd-after-TM inf: ms_Tmgr ms_pimd
> property $id="cib-bootstrap-options" \
>        dc-version="1.0.11-55a5f5be61c367cbd676c2f0ec4f1c62b38223d7" \
>        cluster-infrastructure="Heartbeat" \
>        no-quorum-policy="ignore" \
>        stonith-enabled="false"
> rsc_defaults $id="rsc-options" \
>        migration_threshold="3" \
>        resource-stickiness="100"*
>
> With both nodes up and running, if heartbeat service is stopped on any of
> the nodes, following resources are restarted on the other node:
> mysqld_clone, ms_Rmgr, ms_Tmgr, ms_pimd, ClusterIP
>
> From the Heartbeat debug logs, it seems policy engine is initiating a
> restart operation for the above resources but the reason for the same is not
> clear.
>
> Following are some excerpts from the logs:
>
> "Feb 07 11:06:31 MCG1 pengine: [20534]: info: determine_online_status: Node
> mcg2 is shutting down
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: determine_online_status: Node
> mcg1 is online
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: clone_print:  Master/Slave
> Set: ms_Rmgr
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource Rmgr:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource Rmgr:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource Rmgr:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource Rmgr:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: short_print:      Masters: [
> mcg1 ]
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: short_print:      Slaves: [
> mcg2 ]
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: clone_print:  Master/Slave
> Set: ms_Tmgr
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource Tmgr:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource Tmgr:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource Tmgr:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource Tmgr:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: short_print:      Masters: [
> mcg1 ]
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: short_print:      Slaves: [
> mcg2 ]
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: clone_print:  Master/Slave
> Set: ms_pimd
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource pimd:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource pimd:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource pimd:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource pimd:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: short_print:      Masters: [
> mcg1 ]
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: short_print:      Slaves: [
> mcg2 ]
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: native_print: ClusterIP
>  (ocf::mcg:MCG_VIPaddr_RA):      Started mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: clone_print:  Clone Set:
> EMS_CLONE
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource EMS:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource EMS:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource EMS:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource EMS:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: short_print:      Started: [
> mcg1 mcg2 ]
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: native_print: NDB_VIP
>  (ocf::heartbeat:IPaddr2):       Started mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: native_print: NDB_MGMT
> (ocf::mcg:NDB_MGM_RA):  Started mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: clone_print:  Clone Set:
> mysqld_clone
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource
> mysql:0 active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource
> mysql:0 active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource
> mysql:1 active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource
> mysql:1 active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: short_print:      Started: [
> mcg1 mcg2 ]
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: clone_print:  Clone Set:
> ndbdclone
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource ndbd:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource ndbd:0
> active on mcg1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource ndbd:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_active: Resource ndbd:1
> active on mcg2
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: short_print:      Started: [
> mcg1 mcg2 ]
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource Rmgr:1: preferring current location (node=mcg2, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource Tmgr:1: preferring current location (node=mcg2, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource pimd:1: preferring current location (node=mcg2, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource EMS:1: preferring current location (node=mcg2, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource mysql:1: preferring current location (node=mcg2, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource ndbd:1: preferring current location (node=mcg2, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource Rmgr:0: preferring current location (node=mcg1, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource Tmgr:0: preferring current location (node=mcg1, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource pimd:0: preferring current location (node=mcg1, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource ClusterIP: preferring current location (node=mcg1, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource EMS:0: preferring current location (node=mcg1, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource NDB_VIP: preferring current location (node=mcg1, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource NDB_MGMT: preferring current location (node=mcg1, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource mysql:0: preferring current location (node=mcg1, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: common_apply_stickiness:
> Resource ndbd:0: preferring current location (node=mcg1, weight=100)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Assigning
> mcg1 to Rmgr:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: All nodes
> for resource Rmgr:1 are unavailable, unclean or shutting down (mcg2: 0,
> -1000000)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Could not
> allocate a node for Rmgr:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: native_color: Resource Rmgr:1
> cannot run anywhere
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_color: Allocated 1
> ms_Rmgr instances of a possible 2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: master_color: Rmgr:0 master
> score: 10
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: master_color: Promoting Rmgr:0
> (Master mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: master_color: Rmgr:1 master
> score: 0
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: master_color: ms_Rmgr: Promoted
> 1 instances of a possible 1 to master
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Assigning
> mcg1 to Tmgr:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: All nodes
> for resource Tmgr:1 are unavailable, unclean or shutting down (mcg2: 0,
> -1000000)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Could not
> allocate a node for Tmgr:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: native_color: Resource Tmgr:1
> cannot run anywhere
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_color: Allocated 1
> ms_Tmgr instances of a possible 2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: master_color: Tmgr:0 master
> score: 10
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: master_color: Promoting Tmgr:0
> (Master mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: master_color: Tmgr:1 master
> score: 0
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: master_color: ms_Tmgr: Promoted
> 1 instances of a possible 1 to master
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Assigning
> mcg1 to pimd:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: All nodes
> for resource pimd:1 are unavailable, unclean or shutting down (mcg2: 0,
> -1000000)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Could not
> allocate a node for pimd:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: native_color: Resource pimd:1
> cannot run anywhere
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_color: Allocated 1
> ms_pimd instances of a possible 2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: master_color: pimd:0 master
> score: 10
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: master_color: Promoting pimd:0
> (Master mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: master_color: pimd:1 master
> score: 0
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: master_color: ms_pimd: Promoted
> 1 instances of a possible 1 to master
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Assigning
> mcg1 to ClusterIP
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Assigning
> mcg1 to EMS:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: All nodes
> for resource EMS:1 are unavailable, unclean or shutting down (mcg2: 0,
> -1000000)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Could not
> allocate a node for EMS:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: native_color: Resource EMS:1
> cannot run anywhere
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_color: Allocated 1
> EMS_CLONE instances of a possible 2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Assigning
> mcg1 to NDB_VIP
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Assigning
> mcg1 to NDB_MGMT
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Assigning
> mcg1 to mysql:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: All nodes
> for resource mysql:1 are unavailable, unclean or shutting down (mcg2: 0,
> -1000000)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Could not
> allocate a node for mysql:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: native_color: Resource mysql:1
> cannot run anywhere
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_color: Allocated 1
> mysqld_clone instances of a possible 2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Assigning
> mcg1 to ndbd:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: All nodes
> for resource ndbd:1 are unavailable, unclean or shutting down (mcg2: 0,
> -1000000)
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Could not
> allocate a node for ndbd:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: native_color: Resource ndbd:1
> cannot run anywhere
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_color: Allocated 1
> ndbdclone instances of a possible 2
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: master_create_actions:
> Creating actions for ms_Rmgr
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: master_create_actions:
> Creating actions for ms_Tmgr
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: master_create_actions:
> Creating actions for ms_pimd
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: stage6: Scheduling Node mcg2
> for shutdown
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: Pairing
> Rmgr:0 with Tmgr:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: find_compatible_child: Can't
> pair Tmgr:1 with ms_Rmgr
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: No match
> found for Tmgr:1 (0)
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: clone_rsc_order_lh: Inhibiting
> Tmgr:1 from being active
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Could not
> allocate a node for Tmgr:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: Pairing
> Tmgr:0 with Rmgr:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: Pairing
> Tmgr:1 with Rmgr:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: Pairing
> Tmgr:0 with pimd:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: find_compatible_child: Can't
> pair pimd:1 with ms_Tmgr
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: No match
> found for pimd:1 (0)
> Feb 07 11:06:31 MCG1 pengine: [20534]: info: clone_rsc_order_lh: Inhibiting
> pimd:1 from being active
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: native_assign_node: Could not
> allocate a node for pimd:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: Pairing
> pimd:0 with Tmgr:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: Pairing
> pimd:1 with Tmgr:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: Pairing
> Rmgr:0 with mysql:0
> Feb 07 11:06:31 MCG1 pengine: [20534]: debug: clone_rsc_order_lh: Pairing
> Rmgr:1 with mysql:1
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Restart resource
> Rmgr:0      (Master mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Stop    resource
> Rmgr:1      (mcg2)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Restart resource
> Tmgr:0      (Master mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Stop    resource
> Tmgr:1      (mcg2)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Restart resource
> pimd:0      (Master mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Stop    resource
> pimd:1      (mcg2)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Restart resource
> ClusterIP   (Started mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Leave   resource
> EMS:0       (Started mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Stop    resource
> EMS:1       (mcg2)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Leave   resource
> NDB_VIP     (Started mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Leave   resource
> NDB_MGMT    (Started mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Restart resource
> mysql:0     (Started mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Stop    resource
> mysql:1     (mcg2)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Leave   resource
> ndbd:0      (Started mcg1)
> Feb 07 11:06:31 MCG1 pengine: [20534]: notice: LogActions: Stop    resource
> ndbd:1      (mcg2)
> "
> Thanks in advance.
>
> Regards
> Neha Chatrath
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>