<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Times New Roman; font-size: 12pt; color: #000000'>Hi Andrew,<div><br></div><div>I've installed pacemaker and updated my ha.cf on the quorum node, removing ccm and replacing it with the pacemaker respawn line. You're correct - migration happens quickly and does not need to wait for these timeouts. <span style="font-size: 12pt; ">I then reverted crmd-integration-timeout to its default 3m value.</span></div><div><span style="font-size: 12pt; "><br></span></div><div><span style="font-size: 12pt; ">Thanks,</span></div><div><span style="font-size: 12pt; "><br></span></div><div><span style="font-size: 12pt; ">Andrew</span></div><div><br><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b><span>"Andrew Beekhof" <<a class="smarterwiki-linkify" href="mailto:andrew@beekhof.net" title="[GMCP] Compose a new mail to andrew@beekhof.net" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=andrew@beekhof.net','Compose new message','width=640,height=480');return false">andrew@beekhof.net</a>></span><br><b>To: </b><span>"The Pacemaker cluster resource manager" <<a class="smarterwiki-linkify" href="mailto:pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">pacemaker@oss.clusterlabs.org</a>></span><br><b>Sent: </b>Monday, February 27, 2012 5:25:46 AM<br><b>Subject: </b>Re: [Pacemaker] Configuring 3rd Node as Quorum Node in 2 Node Cluster<br><br>It looks like we're waiting for the other node to respond, which it wont do.<br>Is running pacemaker on the other node but with standby=true an option for you?<br><br><span>On Sat, Feb 25, 2012 at 6:25 AM, Andrew Martin <<a class="smarterwiki-linkify" href="mailto:amartin@xes-inc.com" title="[GMCP] Compose a new mail to amartin@xes-inc.com" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=amartin@xes-inc.com','Compose new message','width=640,height=480');return false">amartin@xes-inc.com</a>> wrote:</span><br>> Hi Andreas,<br>><br>> Thanks, adding "respawn hacluster /usr/lib/heartbeat/ccm" to ha.cf worked.<br>> Since quorum-node is in standby, it shows up as "OFFLINE (standby)" in<br>> crm_mon. It seems that "cl_status nodestatus quorum-node" always returns<br>> "active", even if heartbeat is stopped on the quorum node. However,<br>> "cl_status hblinkstatus quorum-node br0" can correctly detect if heartbeat<br>> is down on quorum-node so I can use that to check its connectivity.<br>><br>> I was able to successfully test resources automatically stopping once quorum<br>> was lost. I did this by shutting down node2 so that only node1 and<br>> quorum-node remained. I then stopped heartbeat on quorum-node, which<br>> resulted in node1 losing quorum and the resources stopping (as expected).<br>> After starting heartbeat on quorum-node again, node1 reestablished quorum<br>> within about 1 minute. However, it took significantly longer (around 18<br>> minutes) for the resources on node1 to start again. Looking through the<br>> logs, I discovered that this is because of the values of<br>> the cluster-recheck-interval (displayed as "PEngine Recheck Timer" in the<br>> logs) and crmd-integration-timeout (displayed as "Integration Timer" in the<br>> logs) properties. Here's the sequence of events as I understand it:<br>> 1. quorum is reestablished<br>> 2. the cluster-recheck-interval timer pops, sees that quorum has been<br>> reestablished, and schedules crmd-integration-timeout to run<br>> 3. after crmd-integration-timeout's timeout period, it pops and also sees<br>> that quorum has been restablished and thus starts the resources<br>><br>> Based on this, the maximum wait time for resources to start once quorum has<br>> been reestablished is the value of cluster-recheck-interval plus the value<br>> of crmd-integration-timeout, or (15m + 3m). I have confirmed this value<br>> through several runs of this test. This seems like a very long time to me,<br>> so I adjusted both of these values down to 1m. Running the test again I was<br>> able to confirm that the resources started 2m after quorum was<br>> reestablished:<br>> ## quorum reestablished<br>> 12:35:31 node1 ccm: [27015]: debug: quorum plugin: majority<br>> 12:35:31 node1 ccm: [27015]: debug: cluster:linux-ha, member_count=2,<br>> member_quorum_votes=200<br>> 12:35:31 node1 ccm: [27015]: debug: total_node_count=3,<br>> total_quorum_votes=300<br>> 12:35:31 node1 crmd: [27020]: info: crmd_ccm_msg_callback: Quorum<br>> (re)attained after event=NEW MEMBERSHIP (id=14)<br>> 12:35:31 node1 crmd: [27020]: info: crm_update_quorum: Updating quorum<br>> status to true (call=366)<br>> ## cluster-recheck-interval pops, schedules crmd-integration-timeout to run<br>> after its timout<br>> 12:36:18 node1 crmd: [27020]: info: crm_timer_popped: PEngine Recheck Timer<br>> (I_PE_CALC) just popped (60000ms)<br>> 12:36:18 node1 crmd: [27020]: info: do_state_transition: State transition<br>> S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED<br>> origin=crm_timer_popped ]<br>> 12:36:18 node1 crmd: [27020]: info: do_state_transition: Progressed to state<br>> S_POLICY_ENGINE after C_TIMER_POPPED<br>> ## crmd-integration-timeout runs, starts the resources<br>> 12:37:18 node1 crmd: [27020]: ERROR: crm_timer_popped: Integration Timer<br>> (I_INTEGRATED) just popped in state S_INTEGRATION! (60000ms)<br>> 12:37:18 node1 crmd: [27020]: info: crm_timer_popped: Welcomed: 1,<br>> Integrated: 1<br>> 12:37:18 node1 crmd: [27020]: info: do_state_transition: State transition<br>> S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_TIMER_POPPED<br>> origin=crm_timer_popped ]<br>> 12:37:18 node1 crmd: [27020]: WARN: do_state_transition: Progressed to state<br>> S_FINALIZE_JOIN after C_TIMER_POPPED<br>> 12:37:20 node1 crmd: [27020]: info: do_dc_join_final: Ensuring DC, quorum<br>> and node attributes are up-to-date<br>> 12:37:20 node1 crmd: [27020]: info: crm_update_quorum: Updating quorum<br>> status to true (call=379)<br>> 12:37:20 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_drbd_r0:1#011(Stopped)<br>> 12:37:20 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_drbd_r1:1#011(Stopped)<br>> 12:37:20 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_drbd_r2:1#011(Stopped)<br>> 12:37:20 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_libvirt-bin:1#011(Stopped)<br>> 12:37:20 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_libvirt-bin:2#011(Stopped)<br>> 12:37:21 node1 crmd: [27020]: notice: run_graph: Transition 77 (Complete=24,<br>> Pending=0, Fired=0, Skipped=3, Incomplete=0,<br>> Source=/var/lib/pengine/pe-input-526.bz2): Stopped<br>> 12:37:21 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_drbd_r0:1#011(Stopped)<br>> 12:37:21 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_drbd_r1:1#011(Stopped)<br>> 12:37:21 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_drbd_r2:1#011(Stopped)<br>> 12:37:21 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_libvirt-bin:1#011(Stopped)<br>> 12:37:21 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_libvirt-bin:2#011(Stopped)<br>> 12:37:23 node1 lrmd: [27017]: info: RA output: (p_vm:start:stdout) Domain<br>> MyVM started<br>> 12:38:23 node1 crmd: [27020]: info: crm_timer_popped: PEngine Recheck Timer<br>> (I_PE_CALC) just popped (60000ms)<br>> 12:38:23 node1 crmd: [27020]: info: do_state_transition: State transition<br>> S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED<br>> origin=crm_timer_popped ]<br>> 12:38:23 node1 crmd: [27020]: info: do_state_transition: Progressed to state<br>> S_POLICY_ENGINE after C_TIMER_POPPED<br>> 12:38:23 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_drbd_r0:1#011(Stopped)<br>> 12:38:23 node1 pengine: [29916]: notice: LogActions: Leave<br>> p_drbd_r1:1#011(Stopped)<br>><br>> I've attached the full logs from this time period in addition to the excerpt<br>> above.<br>><br>> Is there a better way to trigger the starting of resources once quorum has<br>> been reestablished? Or is modifying these two properties a good way of doing<br>> it?<br>><br>> Thanks,<br>><br>> Andrew<br>><br>> ________________________________<br><span>> From: "Andreas Kurz" <<a class="smarterwiki-linkify" href="mailto:andreas@hastexo.com" title="[GMCP] Compose a new mail to andreas@hastexo.com" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=andreas@hastexo.com','Compose new message','width=640,height=480');return false">andreas@hastexo.com</a>></span><br><span>> To: <a class="smarterwiki-linkify" href="mailto:pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">pacemaker@oss.clusterlabs.org</a></span><br>> Sent: Friday, February 24, 2012 7:26:59 AM<br>> Subject: Re: [Pacemaker] Configuring 3rd Node as Quorum Node<br>> in 2 Node Cluster<br>><br>><br>> Hello,<br>><br>> On 02/23/2012 03:59 PM, Andrew Martin wrote:<br>>> I set up the 3rd node ("quorum") yesterday by only installing heartbeat,<br>>> not pacemaker. Is pacemaker necessary as well? I commented out the<br>>> following lines in its ha.cf since it is always going to be running in<br>>> standby:<br>>> autojoin none<br>>> mcast eth0 239.0.0.43 694 1 0<br>>> bcast eth0<br>>> warntime 5<br>>> deadtime 15<br>>> initdead 60<br>>> keepalive 2<br>>> node node1<br>>> node node2<br>>> node quorum<br>>> #crm respawn<br>>> #respawn hacluster /usr/lib/heartbeat/dopd<br>>> #apiauth dopd gid=haclient uid=hacluster<br>>><br>><br>> Hmm ... IIRC I had to enable ccm in ha.cf on the third node during my<br>> last heartbeat tests to enable a quorum node:<br>><br>> respawn hacluster ccm<br>><br>>> Since this quorum node only has a single ethernet interface, can it be<br>>> used for both the mcast and bcast parameters? How are both the multicast<br>>> and broadcast pathways used for node communication? After saving these<br>>> parameters and reloading heartbeat on all nodes, the "quorum" node is<br>>> listed as offline in the cluster. Is there something missing in my<br>>> configuration that is preventing it from communicating with the rest of<br>>> the cluster?<br>><br>> cl_status and hbclient should give you some membership information, and<br>> there should be some log entries on the nodes running Pacemaker. I don't<br>> think the node will show up as ONLINE in crm_mon if no Pacemaker is<br>> running there.<br>><br>> You only need the communication settings you also used for node1/2 on<br>> the shared network ... so only the mcast directive is needed/possible on<br>> node3.<br>><br>>><br>>> Also, another more general question about the failover - node1 and node2<br>>> are each connected to the shared network over br0 and connected directly<br>>> to each other with a crossover cable over br1:<br>>> -------------------- ----------<br>>> | Shared Network |------| quorum |<br>>> -------------------- ----------<br>>> |<br>>> br0 / \ br0<br>>> / \<br>>> --------- ---------<br>>> | node1 | --------- | node2 |<br>>> --------- br1 ---------<br>>><br>>> The corresponding configuration in ha.cf is<br>>> autojoin none<br>>> mcast br0 239.0.0.43 694 1 0<br>>> bcast br1<br>>> warntime 5<br>>> ....<br>>><br>>> If br0 to one of the nodes were to be cut when the "quorum" node was<br>>> down, would they still be able to communicate over br1 (e.g. to maintain<br>>> quorum between themselves and fail over to the other node that still has<br>>> an active br0)?<br>><br>> as long as node1/2 can communicate the cluster has quorum, to fail over<br>> resources to the node with best connectivity: configure ping resource<br>> agent and constraints, there is a chapter in "Pacemake Explained" on<br><span>> clusterlabs.org ... <a class="smarterwiki-linkify" href="http://goo.gl/x7dwK" target="_blank">http://goo.gl/x7dwK</a></span><br>><br>> Regards,<br>> Andreas<br>><br>> --<br>> Need help with Pacemaker?<br><span>> <a class="smarterwiki-linkify" href="http://www.hastexo.com/now" target="_blank">http://www.hastexo.com/now</a></span><br>><br>>><br>>> Thanks,<br>>><br>>> Andrew<br>>><br>>> ------------------------------------------------------------------------<br><span>>> *From: *"Andreas Kurz" <<a class="smarterwiki-linkify" href="mailto:andreas@hastexo.com" title="[GMCP] Compose a new mail to andreas@hastexo.com" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=andreas@hastexo.com','Compose new message','width=640,height=480');return false">andreas@hastexo.com</a>></span><br><span>>> *To: <a class="smarterwiki-linkify" href="mailto:*pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to *pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=*pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">*pacemaker@oss.clusterlabs.org</a></span><br>>> *Sent: *Monday, January 23, 2012 1:53:27 PM<br>>> *Subject: *Re: [Pacemaker] Configuring 3rd Node as Quorum Node in<br>>> 2 Node Cluster<br>>><br>>> On 01/23/2012 03:36 PM, Andrew Martin wrote:<br>>>> I think I will configure the 3rd (quorum) node in standby mode. In the<br>>>> near future I am looking into setting up 2 additional clusters (each of<br>>>> these are also 2-node clusters) and would like to use this same server<br>>>> as the quorum node for those clusters as well. Is this possible? If so,<br>>>> how do I have to configure heartbeat (or multiple instances of<br>>>> heartbeat) to join multiple clusters at once and act as the quorum node<br>>>> in each?<br>>><br>>> No, multiple heartbeat instances per node are not supported ... but why<br>>> not creating minimal VM instances ... though not to minimal, as you have<br>>> a good chance that theses standby instances become DC role.<br>>><br>>> Regards,<br>>> Andreas<br>>><br>>> --<br>>> Need help with Pacemaker?<br><span>>> <a class="smarterwiki-linkify" href="http://www.hastexo.com/now" target="_blank">http://www.hastexo.com/now</a></span><br>>><br>>>><br>>>> Thanks,<br>>>><br>>>> Andrew<br>>>><br>>>> ------------------------------------------------------------------------<br><span>>>> *From: *"Andreas Kurz" <<a class="smarterwiki-linkify" href="mailto:andreas@hastexo.com" title="[GMCP] Compose a new mail to andreas@hastexo.com" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=andreas@hastexo.com','Compose new message','width=640,height=480');return false">andreas@hastexo.com</a>></span><br><span>>>> *To: <a class="smarterwiki-linkify" href="mailto:*pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to *pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=*pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">*pacemaker@oss.clusterlabs.org</a></span><br>>>> *Sent: *Friday, January 13, 2012 6:35:48 AM<br>>>> *Subject: *Re: [Pacemaker] Configuring 3rd Node as Quorum Node in 2<br>>>> Node Cluster<br>>>><br>>>> On 01/13/2012 12:32 PM, Ivan Savčić | Epix wrote:<br>>>>> On 1/11/2012 8:28 AM, Florian Haas wrote:<br>>>>>> Another option would be to permanently run the 3rd node in standby<br>>>>>> mode.<br>>>>><br>>>>> Just wondering, wouldn't the standby mode prevent that node from<br>>>>> performing the fencing actions? Also, can it act as DC then?<br>>>><br>>>> It would run no resources (including stonith resources) but can be the<br>>>> DC.<br>>>><br>>>> Another option for running a "pure" quorum node would be to only start<br>>>> CCM but not pacemaker ... though that setup looks quite strange e.g. in<br>>>> crm_mon output ....<br>>>><br>>>> Regards,<br>>>> Andreas<br>>>><br>>>> --<br>>>> Need help with Pacemaker?<br><span>>>> <a class="smarterwiki-linkify" href="http://www.hastexo.com/now" target="_blank">http://www.hastexo.com/now</a></span><br>>>><br>>>>><br>>>>> Thanks,<br>>>>> Ivan<br>>>>><br>>>>> _______________________________________________<br><span>>>>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">Pacemaker@oss.clusterlabs.org</a></span><br><span>>>>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>>>>><br><span>>>>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a></span><br><span>>>>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>>>>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></span><br>>>><br>>>><br>>>><br>>>><br>>>> _______________________________________________<br><span>>>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">Pacemaker@oss.clusterlabs.org</a></span><br><span>>>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>>>><br><span>>>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a></span><br><span>>>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>>>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></span><br>>>><br>>>><br>>>><br>>>> _______________________________________________<br><span>>>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">Pacemaker@oss.clusterlabs.org</a></span><br><span>>>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>>>><br><span>>>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a></span><br><span>>>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>>>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></span><br>>><br>>><br>>><br>>> _______________________________________________<br><span>>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">Pacemaker@oss.clusterlabs.org</a></span><br><span>>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>>><br><span>>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a></span><br><span>>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></span><br>>><br>>><br>>><br>>> _______________________________________________<br><span>>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">Pacemaker@oss.clusterlabs.org</a></span><br><span>>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>>><br><span>>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a></span><br><span>>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></span><br>><br>><br>><br>> _______________________________________________<br><span>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">Pacemaker@oss.clusterlabs.org</a></span><br><span>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>><br><span>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a></span><br><span>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></span><br>><br>><br>> _______________________________________________<br><span>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">Pacemaker@oss.clusterlabs.org</a></span><br><span>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>><br><span>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a></span><br><span>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></span><br>><br><br>_______________________________________________<br><span>Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false">Pacemaker@oss.clusterlabs.org</a></span><br><span><a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br><br><span>Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a></span><br><span>Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></span><br></div><br></div></div></body></html>