[Pacemaker] Problems with SBD fencing

Jan Christian Kaldestad janck76 at gmail.com
Tue Aug 6 11:49:42 UTC 2013 

In my case this does not work - read my original post. So I wonder if there
is a pacemaker bug (version 1.1.9-2db99f1). Killing pengine and stonithd on
the node which is supposed to "shoot" seems to resolve the problem, though
this is not a solution of course.

I also tested two separate stonith resources, one on each node. This
stonith'ing works fine with this configuration. Is there somehing "wrong"
about doing it this way?


Best regards
Jan






On Tue, Aug 6, 2013 at 11:50 AM, Dejan Muhamedagic <dejanmm at fastmail.fm>wrote:

> Hi,
>
> On Tue, Aug 06, 2013 at 11:22:56AM +0200, Andreas Mock wrote:
> > Hi Dejan,
> >
> > can you explain how the SDB agent works, when this resource
> > is running on exactly that node which has to be stonithed?
>
> It's actually in the hands of the resource manager to take care
> of that. The pacemaker is going to start the stonith resource in
> case another node is to be fenced.
>
> Thanks,
>
> Dejan
>
> > Thank you in advance.
> >
> > Best regards
> > Andreas Mock
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dejan Muhamedagic [mailto:dejanmm at fastmail.fm]
> > Gesendet: Dienstag, 6. August 2013 11:15
> > An: The Pacemaker cluster resource manager
> > Betreff: Re: [Pacemaker] Problems with SBD fencing
> >
> > Hi,
> >
> > On Thu, Aug 01, 2013 at 07:58:55PM +0200, Jan Christian Kaldestad wrote:
> > > Thanks for the explanation. But I'm quite confused about the SBD
> stonith
> > > resource configuration, as the SBD fencing wiki clearly states:
> > > "The sbd agent does not need to and should not be cloned. If all of
> your
> > > nodes run SBD, as is most likely, not even a monitor action provides a
> > real
> > > benefit, since the daemon would suicide the node if there was a
> problem. "
> > >
> > > and also this thread
> > >
> >
> http://oss.clusterlabs.org/pipermail/pacemaker/2012-March/013507.htmlmention
> > > that there should be only one SBD resource configured.
> > >
> > > Can someone please clarify? Should I configure 2 separate SBD
> resources,
> > > one for each cluster node?
> >
> > No. One sbd resource is sufficient.
> >
> > Thanks,
> >
> > Dejan
> >
> > >
> > > --
> > > Best regards
> > > Jan
> > >
> > >
> > > On Thu, Aug 1, 2013 at 6:47 PM, Andreas Mock <andreas.mock at web.de>
> wrote:
> > >
> > > > Hi Jan,****
> > > >
> > > > ** **
> > > >
> > > > first of all I don't know the SBD-Fencing-Infrastructure (just read
> > the***
> > > > *
> > > >
> > > > article linked by you). But as far as I understand the "normal"
> > fencing***
> > > > *
> > > >
> > > > (initiated on behalf of pacemaker) is done in the following way.****
> > > >
> > > > ** **
> > > >
> > > > SBD fencing resoure (agent) is writing a request for self-stonithing
> > into*
> > > > ***
> > > >
> > > > one or more SBD partitions where the SBD-daemon is listening and
> > hopefully
> > > > ****
> > > >
> > > > reacting on.****
> > > >
> > > > So, I'm pretty sure (without knowing) that you have to configure
> the****
> > > >
> > > > stonith agent in a way that pacemaker knows howto talk to the stonith
> > agent
> > > > ****
> > > >
> > > > to kill a certain cluster node.****
> > > >
> > > > What is the problem in you scenario: The agent which should be
> > contacted**
> > > > **
> > > >
> > > > to stonith the node2 is/was running on node2 and can't be connected
> > > > anymore.****
> > > >
> > > > ** **
> > > >
> > > > Because of that stonith agent configuration is most of the times done
> > the*
> > > > ***
> > > >
> > > > following way in a two node cluster:****
> > > >
> > > > On every node runs a stonith agent. The stonith agent is configured
> > to****
> > > >
> > > > stonith the OTHER node. You have to be sure that this is technically
> > ****
> > > >
> > > > always possible.****
> > > >
> > > > This can be achieved with resource clones or - which is IMHO simpler
> -
> > in
> > > > ****
> > > >
> > > > a 2-node-environment with two stonith resources and a negative
> > colocation*
> > > > ***
> > > >
> > > > constraint.****
> > > >
> > > > ** **
> > > >
> > > > As far as I know there is also a self-stonith-safty-belt
> implemented****
> > > >
> > > > in a way that a stonith agent on a node to be shot is never
> > contacted.****
> > > >
> > > > (Do I remember correct?)****
> > > >
> > > > ** **
> > > >
> > > > I'm sure this may solve your problem.****
> > > >
> > > > ** **
> > > >
> > > > Best regards****
> > > >
> > > > Andreas Mock****
> > > >
> > > > ** **
> > > >
> > > > ** **
> > > >
> > > > *Von:* Jan Christian Kaldestad [mailto:janck76 at gmail.com]
> > > > *Gesendet:* Donnerstag, 1. August 2013 15:46
> > > > *An:* pacemaker at oss.clusterlabs.org
> > > > *Betreff:* [Pacemaker] Problems with SBD fencing****
> > > >
> > > > ** **
> > > >
> > > > Hi,****
> > > >
> > > >
> > > > I am evaluating the SLES HA Extension 11 SP3 product. The cluster
> > > > consists of 2-nodes (active/passive), using SBD stonith resource on a
> > > > shared SAN disk. Configuration according to
> > > > http://www.linux-ha.org/wiki/SBD_Fencing****
> > > >
> > > > The SBD daemon is running on both nodes, and the stontih resource
> > (defined
> > > > as primitive) is running on one node only.
> > > > There is also a monitor operation for the stonith resource
> > > > (interval=36000, timeout=20)****
> > > >
> > > > I am having some problems getting failover/fencing to work as
> expected
> > in
> > > > the following scenario:
> > > > - Node 1 is running the resources that I created (except stonith)
> > > > - Node 2 is running the stonith resource
> > > > - Disconnect Node 2 from the network by bringing the interface down
> > > > - Node 2 status changes to UNCLEAN (offline), but the stonith
> resource
> > > > does not switch over to Node 1 and Node 2 does not reboot as I would
> > expect.
> > > > - Checking the logs on Node 1, I notice the following:
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: pe_fence_node:
> > Node
> > > > slesha1n2i-u will be fenced because the node is no longer part of the
> > > > cluster
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning:
> > > > determine_online_status: Node slesha1n2i-u is unclean
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: custom_action:
> > > > Action stonith_sbd_stop_0 on slesha1n2i-u is unrunnable (offline)
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: stage6:
> > Scheduling
> > > > Node slesha1n2i-u for STONITH
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: LogActions:
> Move
> > > > stonith_sbd   (Started slesha1n2i-u -> slesha1n1i-u)
> > > >  ...
> > > >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice: te_fence_node:
> > > > Executing reboot fencing operation (24) on slesha1n2i-u
> (timeout=60000)
> > > >  Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:   notice:
> > handle_request:
> > > > Client crmd.8916.3144546f wants to fence (reboot) 'slesha1n2i-u' with
> > > > device '(any)'
> > > >  Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:   notice:
> > > > initiate_remote_stonith_op: Initiating remote operation reboot for
> > > > slesha1n2i-u: 8c00ff7b-2986-4b2a-8b4a-760e8346349b (0)
> > > >  Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:    error:
> > remote_op_done:
> > > > Operation reboot of slesha1n2i-u by slesha1n1i-u for
> > > > crmd.8916 at slesha1n1i-u.8c00ff7b: No route to host
> > > >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
> > > > tengine_stonith_callback: Stonith operation
> > > > 3/24:3:0:8a0f32b2-f91c-4cdf-9cee-1ba9b6e187ab: No route to host
> (-113)
> > > >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
> > > > tengine_stonith_callback: Stonith operation 3 for slesha1n2i-u failed
> > (No
> > > > route to host): aborting transition.
> > > >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
> > > > tengine_stonith_notify: Peer slesha1n2i-u was not terminated
> > > > (st_notify_fence) by slesha1n1i-u for slesha1n1i-u: No route to host
> > > > (ref=8c00ff7b-2986-4b2a-8b4a-760e8346349b) by client crmd.8916
> > > >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice: run_graph:
> > Transition
> > > > 3 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0,
> > > > Source=/var/lib/pacemaker/pengine/pe-warn-15.bz2): Stopped
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice:
> unpack_config: On
> > > > loss of CCM Quorum: Ignore
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: pe_fence_node:
> > Node
> > > > slesha1n2i-u will be fenced because the node is no longer part of the
> > > > cluster
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning:
> > > > determine_online_status: Node slesha1n2i-u is unclean
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: custom_action:
> > > > Action stonith_sbd_stop_0 on slesha1n2i-u is unrunnable (offline)
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: stage6:
> > Scheduling
> > > > Node slesha1n2i-u for STONITH
> > > >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: LogActions:
> Move
> > > > stonith_sbd   (Started slesha1n2i-u -> slesha1n1i-u)
> > > >  ...
> > > >  Aug  1 12:00:02 slesha1n1i-u crmd[8916]:   notice:
> > too_many_st_failures:
> > > > Too many failures to fence slesha1n2i-u (11), giving up
> > > >  ****
> > > >
> > > >  ****
> > > >
> > > >
> > > > - Then I bring Node 1 online again and start the cluster service,
> > checking
> > > > logs:
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] CLM
> > CONFIGURATION
> > > > CHANGE
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] New
> > Configuration:
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0)
> > ip(x.x.x.x)
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Left:
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members
> Joined:
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > > > pcmk_peer_update: Transitional membership event on ring 376: memb=1,
> > new=0,
> > > > lost=0
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > pcmk_peer_update: memb: slesha1n1i-u 168824371
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] CLM
> > CONFIGURATION
> > > > CHANGE
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] New
> > Configuration:
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0)
> > ip(x.x.x.x)
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0)
> > ip(y.y.y.y)
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Left:
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members
> Joined:
> > > >  Aug  1 12:31:13 slesha1n1i-u cib[8911]:   notice:
> ais_dispatch_message:
> > > > Membership 376: quorum acquired
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0)
> > ip(y.y.y.y)
> > > >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
> > ais_dispatch_message:
> > > > Membership 376: quorum acquired
> > > >  Aug  1 12:31:13 slesha1n1i-u cib[8911]:   notice:
> > crm_update_peer_state:
> > > > crm_update_ais_node: Node slesha1n2i-u[168824372] - state is now
> member
> > > > (was lost)
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > > > pcmk_peer_update: Stable membership event on ring 376: memb=2, new=1,
> > lost=0
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > update_member: Node 168824372/slesha1n2i-u is now: member
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > pcmk_peer_update: NEW:  slesha1n2i-u 168824372
> > > >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
> > crm_update_peer_state:
> > > > crm_update_ais_node: Node slesha1n2i-u[168824372] - state is now
> member
> > > > (was lost)
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > pcmk_peer_update: MEMB: slesha1n1i-u 168824371
> > > >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
> > peer_update_callback:
> > > > Node return implies stonith of slesha1n2i-u (action 24) completed
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > pcmk_peer_update: MEMB: slesha1n2i-u 168824372
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > send_member_notification: Sending membership update 376 to 2 children
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [TOTEM ] A processor
> > joined
> > > > or left the membership and a new membership was formed.
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > update_member: 0x69f2f0 Node 168824372 (slesha1n2i-u) born on: 376
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > send_member_notification: Sending membership update 376 to 2 children
> > > >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
> crm_update_quorum:
> > > > Updating quorum status to true (call=119)
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CPG   ] chosen
> downlist:
> > > > sender r(0) ip(x.x.x.x) ; members(old:1 left:0)
> > > >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [MAIN  ] Completed
> > service
> > > > synchronization, ready to provide service.
> > > >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
> > too_many_st_failures:
> > > > Too many failures to fence slesha1n2i-u (13), giving up
> > > >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
> > too_many_st_failures:
> > > > Too many failures to fence slesha1n2i-u (13), giving up
> > > >  Aug  1 12:31:14 slesha1n1i-u mgmtd: [8917]: info: CIB query: cib
> > > >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > update_member: Node slesha1n2i-u now has process list:
> > > > 00000000000000000000000000151302 (1381122)
> > > >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > send_member_notification: Sending membership update 376 to 2 children
> > > >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > update_member: Node slesha1n2i-u now has process list:
> > > > 00000000000000000000000000141302 (1315586)
> > > >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > send_member_notification: Sending membership update 376 to 2 children
> > > >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > update_member: Node slesha1n2i-u now has process list:
> > > > 00000000000000000000000000101302 (1053442)
> > > >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > > > send_member_notification: Sending membership update 376 to 2 children
> > > >  Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice:
> do_state_transition:
> > > > State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN
> > > > cause=C_HA_MESSAGE origin=route_message ]
> > > >  Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice:
> > too_many_st_failures:
> > > > Too many failures to fence slesha1n2i-u (13), giving up
> > > >  Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice:
> > too_many_st_failures:
> > > > Too many failures to fence slesha1n2i-u (13), giving up****
> > > >
> > > > - Cluster status changes to Online for both nodes, but the stonith
> > > > resource won't start on any of the nodes.
> > > > - Trying to start the resource manually, but no success.
> > > > - Trying to restart the corosync process on Node 1 (rcopenais
> restart),
> > > > but it hangs forever. Checking logs:
> > > >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [SERV  ] Unloading all
> > > > Corosync service engines.
> > > >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > > > pcmk_shutdown: Shuting down Pacemaker
> > > >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > > > stop_child: Sent -15 to mgmtd: [8917]
> > > >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: mgmtd is shutting
> > down
> > > >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: final_crm:
> > client_id=1
> > > > cib_name=live
> > > >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: final_crm:
> > client_id=2
> > > > cib_name=live
> > > >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: debug: [mgmtd] stopped
> > > >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > > > pcmk_shutdown: mgmtd confirmed stopped
> > > >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > > > stop_child: Sent -15 to crmd: [8916]
> > > >  Aug  1 12:42:08 slesha1n1i-u crmd[8916]:   notice: crm_shutdown:
> > > > Requesting shutdown, upper limit is 1200000ms
> > > >  Aug  1 12:42:08 slesha1n1i-u attrd[8914]:   notice:
> > attrd_trigger_update:
> > > > Sending flush op to all hosts for: shutdown (1375353728)
> > > >  Aug  1 12:42:08 slesha1n1i-u attrd[8914]:   notice:
> > attrd_perform_update:
> > > > Sent update 22: shutdown=1375353728
> > > >  Aug  1 12:42:08 slesha1n1i-u crmd[8916]:   notice:
> > too_many_st_failures:
> > > > Too many failures to fence slesha1n2i-u (13), giving up
> > > >  Aug  1 12:42:08 slesha1n1i-u crmd[8916]:  warning: do_log: FSA:
> Input
> > > > I_TE_SUCCESS from abort_transition_graph() received in state
> > S_POLICY_ENGINE
> > > >  Aug  1 12:42:38 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > > > pcmk_shutdown: Still waiting for crmd (pid=8916, seq=6) to
> terminate...
> > > >  Aug  1 12:43:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > > > pcmk_shutdown: Still waiting for crmd (pid=8916, seq=6) to
> > terminate...***
> > > > *
> > > >
> > > > ...****
> > > >
> > > > - Finally I kill the corosync process on Node 1 (killall -9
> corosync),
> > > > then corsoync restarts.
> > > > - Checking status. All resources are up and running on Node 1, and
> the
> > > > stonith resource is running on Node 2 again.****
> > > >
> > > >  ****
> > > >
> > > > I have tested the same scenario several times. Sometimes the fencing
> > > > mechaism works as expected, but other times the stonith resource is
> not
> > > > transferred to Node 1 - as described here. So I need some assistance
> to
> > > > overcome this problem....****
> > > >
> > > >  ****
> > > >
> > > > --
> > > > Best regards
> > > > Jan ****
> > > >
> > > > _______________________________________________
> > > > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > > >
> > > > Project Home: http://www.clusterlabs.org
> > > > Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > Bugs: http://bugs.clusterlabs.org
> > > >
> > > >
> > >
> > >
> > > --
> > > mvh
> > > Jan Christian
> >
> > > _______________________________________________
> > > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > >
> > > Project Home: http://www.clusterlabs.org
> > > Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> >
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>



-- 
mvh
Jan Christian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20130806/8cb48ccc/attachment.htm>

More information about the Pacemaker mailing list