<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>One correction:</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">I removed the "location" constraint and simply went with this:</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div><div> colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master glance-repos-fs-group</div><div> order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start</div><div> order glance-order-fs-after-drbd2 inf: glance-repos-fs-group:stop
ms_drbd:demote ms_drbd:stop glance-repos:stop</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">I called out the stop of DRBD before the stop of LVM. The syslog attached previously is for this configuration.</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">Thanks,</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">Bob</div></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;
"><br></div><div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><font size="2" face="Arial"><hr size="1"><b><span style="font-weight:bold;">From:</span></b> Bob Schatz <bschatz@yahoo.com><br><b><span style="font-weight: bold;">To:</span></b> "pacemaker@oss.clusterlabs.org" <pacemaker@oss.clusterlabs.org><br><b><span style="font-weight: bold;">Sent:</span></b> Wednesday, July 20, 2011 11:32 AM<br><b><span style="font-weight: bold;">Subject:</span></b> [Pacemaker] Fw: Configuration for FS over DRBD over LVM<br></font><br><div id="yiv1439411717"><div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><div style="font-size: 12pt; font-family: times, serif; "><span>I tried another test based on this thread:</span></div><div
style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div style="font-size: 12pt; font-family: times, serif; "><span><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> http://www.gossamer-threads.com/lists/linuxha/pacemaker/65928?search_string=lvm%20drbd;#65928</span></span></div><div style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div style="font-size: 12pt; font-family: times, serif; "><span>I removed the "location" constraint and simply went with this:</span></div><div style="font-size: 12pt; font-family: times, serif; "><span><br></span></div><div><div> colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master glance-repos-fs-group</div><div> order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start</div><div> order glance-order-fs-after-drbd2 inf:
glance-repos-fs-group:stop ms_drbd:demote glance-repos:stop</div></div><div><span><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">The stop actions were called in this order:</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>stop file system</div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>demote DRBD</div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>stop LVM *****</div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>stop DRBD *****</div><div style="font-size:
12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">instead of:</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>stop file system</div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>demote DRBD</div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>stop DRBD ******</div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>stop LVM ******</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">I see these messages in the log which I
believe are debug messages based on reading other threads:</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-start-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-start-end</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-stop-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-stop-end</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-promote-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-promote-end</div><div>
pengine: [21021]: debug: text2task: Unsupported action:
glance-order-fs-after-drbd-1-demote-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-demote-end</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-start-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-start-end</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-stop-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-stop-end</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-stop-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action:
glance-order-fs-after-drbd2-0-stop-end</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-start-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-start-end</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-demote-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-demote-end</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-promote-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-promote-end</div><div> pengine: [21021]: debug: text2task: Unsupported action:
glance-order-fs-after-drbd2-2-stop-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-stop-end</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-start-begin</div><div> pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-start-end</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">I have attached a syslog-pacemaker log of the "/etc/init.d/corosync start" through "/etc/init.d/corosync stop" sequence.</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">Thanks,</div><div style="font-size: 12pt; font-family: times,
serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">Bob</div></div><div><div><br></div></div></span></div><div style="font-size: 12pt; font-family: times, serif; "><div style="font-size: 12pt; font-family: times, serif; "><font size="2" face="Arial">----- Forwarded Message -----<br><b><span style="font-weight:bold;">From:</span></b> Bob Schatz <bschatz@yahoo.com><br><b><span style="font-weight:bold;">To:</span></b> "pacemaker@oss.clusterlabs.org" <pacemaker@oss.clusterlabs.org><br><b><span style="font-weight:bold;">Sent:</span></b> Tuesday, July 19, 2011 4:38 PM<br><b><span style="font-weight:bold;">Subject:</span></b> [Pacemaker]
Configuration for FS over DRBD over LVM<br></font><br><div id="yiv1439411717"><div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-size: 12pt; font-family: times, serif; "><div style="font-size: 12pt; font-family: times, serif; ">Hi,</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">I am trying to configure an FS running on top of DRBD on top of LVM or:</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; "> FS</div><div style="font-size: 12pt; font-family: times, serif; "> |</div><div style="font-size: 12pt; font-family: times, serif; "> DRBD</div><div style="font-size: 12pt; font-family: times, serif; "> |</div><div style="font-size: 12pt; font-family: times, serif; ">
LVM</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">I am using Pacemaker 1.0.8, Ubuntu 10.04 and DRBD 8.3.7.</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">Reviewing all the manuals (Pacemaker Explained 1.0, DRBD 8.4 User Guide, etc) I came up with this Pacemaker configuration:</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>node cnode-1-3-5</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>node cnode-1-3-6</font></div><div><font
class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>primitive glance-drbd ocf:linbit:drbd \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> params drbd_resource="glance-repos-drbd" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> op start interval="0" timeout="240" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times
new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> op stop interval="0" timeout="100" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> op monitor interval="59s" role="Master" timeout="30s" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> op monitor interval="61s" role="Slave" timeout="30s"</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>primitive glance-fs ocf:heartbeat:Filesystem
\</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"> <span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> params device="/dev/drbd1" directory="/glance-mount" fstype="ext4" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"> <span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> op start interval="0" timeout="60" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"> <span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> op monitor interval="60" timeout="60" OCF_CHECK_LEVEL="20" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"> <span
class="yiv1439411717Apple-tab-span" style="white-space:pre;">
</span> op stop interval="0" timeout="120"</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>primitive glance-repos ocf:heartbeat:LVM \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> params volgrpname="glance-repos" exclusive="true" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> op start interval="0" timeout="30" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span"
style="white-space:pre;"> </span> op stop interval="0" timeout="30"</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>group
glance-repos-fs-group glance-fs</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>ms ms_drbd glance-drbd \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> meta master-node-max="1" clone-max="2" clone-node-max="1" globally-unique="false" notify="true" target-role="Master"</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>location drbd_on_node1 ms_drbd \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span"
style="white-space:pre;">
</span> rule $id="drbd_on_node1-rule" $role="Master" 100: #uname eq
cnode-1-3-5</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>property $id="cib-bootstrap-options" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;">
</span>
dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd"
\</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> cluster-infrastructure="openais" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> expected-quorum-votes="1" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> stonith-enabled="false" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>
no-quorum-policy="ignore" \</font></div><div><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times,
serif"><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span> last-lrm-refresh="1310768814"</font></div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">On one node, things come up cleanly. In fact, debug messages in the agent show that the start() functions for the agent are called and exited in order (LVM start, DRBD start and Filesystem start).</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">The problem occurs when I do "/etc/init.d/corosync stop" on a single node. What happens is that the stop() functions are called in this order:</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;">
</span>1. LVM stop</div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>2. Filesystem stop</div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>3. DRBD stop</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">What I have tried:</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;"> </span>1. I tried setting the score of the order to "500" assuming that this would mean the colocation rule would hit first. Still the same problem.</div><div style="font-size: 12pt; font-family: times, serif; "><span class="yiv1439411717Apple-tab-span" style="white-space:pre;">
</span>2. I tried leaving off the ":start" and ":promote" options on the "order" line. The stop order was still LVM, Filesystem, and DRBD</div><div><span class="yiv1439411717Apple-tab-span" style="font-size: 12pt; white-space: pre; font-family: times, serif; "> </span><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif" size="3">3. I tried adding another colocation rule "</font><font class="yiv1439411717Apple-style-span" face="'times new roman', 'new york', times, serif">colocation coloc-rule-w-master2 inf: ms_drbd:Master glance-repos-fs-group" to tie glance-repos-fs-group to the same
node as DRBD. Stop still had the same issue. I assume that I will still need this rule when I add a second node to the test.</font></div><div><br></div><div style="font-size: 12pt; font-family: times, serif; ">Any suggestions would be appreciated.</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">A side note, the reason I have a group for the file system is that I would like to add an application and IP address to the group once I get this working. Also, the reason I have LVM under DRBD is that I want to be able to grow the LVM volume as needed and then expand the DRBD volume.</div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; "><br></div><div style="font-size: 12pt; font-family: times, serif; ">Thanks in advance,</div><div style="font-size: 12pt; font-family: times, serif;
"><br></div><div style="font-size: 12pt; font-family: times, serif; ">Bob</div></div></div></div><br>_______________________________________________<br>Pacemaker mailing list: <a rel="nofollow" ymailto="mailto:Pacemaker@oss.clusterlabs.org" target="_blank" href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br><a rel="nofollow" target="_blank" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br><br>Project Home: <a rel="nofollow" target="_blank" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a><br>Getting started: <a rel="nofollow" target="_blank" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>Bugs: <a rel="nofollow" target="_blank"
href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br><br><br></div></div></div></div><br>_______________________________________________<br>Pacemaker mailing list: <a ymailto="mailto:Pacemaker@oss.clusterlabs.org" href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br><a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br><br>Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker"
target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br><br>_______________________________________________<br>Pacemaker mailing list: <a ymailto="mailto:Pacemaker@oss.clusterlabs.org" href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br><a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br><br>Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br><br><br></div></div></div></body></html>