<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.StileMessaggioDiPostaElettronica17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 2.0cm 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=IT link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Hi,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>It's my first try at asking for help on a mailing list, I hope I'll not make netiquette mistakes. I really could use some help on SBD, here's my scenario:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>I have three clusters with a similar configuration: two physical servers with a fibre channel shared storage, 4 resources (ip address, ext3 filesystem, oracle listener, oracle database) configured in a group, and external\SBD as stonith device. Operating system, is SLES 11 Sp1, cluster components come from the SLES Sp1 HA package and are these versions:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>openais: 1.1.4-5.6.3<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>pacemaker: 1.1.5-5.9.11.1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>resource-agents: 3.9.3-0.4.26.1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>cluster-glue: 1.0.8-0.4.4.1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>corosync: 1.3.3-0.3.1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>csync2: 1.34-0.2.39<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Each one of the three clusters will work fine for a couple of days, then both servers of one of the clusters at the same time will start the SBD "WARN: Latency: No liveness for" countdown and restart. It happens at different hours, and during different servers load (even at night, when servers are close to 0% load). No two clusters have ever went down at the same time. Their syslog is superclean, the only warning messages before the reboots are the ones telling the SBD liveness countdown. The SAN department can’t see anything wrong on their side, the SAN is used by many other servers, no-one seems to be experiencing similar problems.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Hardware<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Cluster 1 and Cluster 2: two IBM blades, QLogic QMI2582 (one card, two ports), Brocade blade center FC switch, SAN switch, HP P9500 SAN <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Cluster 3: two IBM x3650, QLogic QLE2560 (two cards per server), SAN switch, HP P9500 SAN<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Each cluster have a 50GB LUN on the HP P9500 SAN (the SAN is in common, the LUNs are different): partition 1 (7.8 MB) for SBD, partition 2 (49.99 GB) for Oracle on ext3<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>What I have done so far:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>- introduced options qla2xxx ql2xmaxqdepth=16 qlport_down_retry=1 ql2xloginretrycount=5 ql2xextended_error_logging=1 in /etc/modprobe.conf.local (and mkinitrd and restarted the servers)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>- verified with the SAN department that the Qlogic firmware of my HBAs is compliant with their needs<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>- configured multipath.conf as per HP specifications for the OPEN-V type of SAN<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>- verified multipathd is working as expected, shutting down one port at a time, links stay up on the other port, and then shutting down both, cluster switches on the other node<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>- configured SBD to use the watchdog device (softdog), and the first partition of the LUN, and all relevant tests confirm SBD is working as expected (list, dump, message test, message exit, killing the SBD process the server reboots), here's my /etc/sysconfig/SBD<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>server1:~ # cat /etc/sysconfig/SBD<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>SBD_DEVICE="/dev/mapper/san_part1"<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>SBD_OPTS="-W"<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>- enhanced (x2) the default values for Timeout (watchdog) and Timeout (msgwait), setting them at 10 and 20, while Stonith Timeout is 60s<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>server1:~ # SBD -d /dev/mapper/san_part1 dump<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>==Dumping header on disk /dev/mapper/san_part1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Header version : 2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Number of slots : 255<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Sector size : 512<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Timeout (watchdog) : 10<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Timeout (allocate) : 2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Timeout (loop) : 1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Timeout (msgwait) : 20<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>==Header on disk /dev/mapper/san_part1 is dumped<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>I’ve even tested with 60 and 120 for Timeout (watchdog) and Timeout (msgwait), when the problem happened again the serves went all through the 60 seconds delay countdown to reboot.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Borrowing the idea from here http://www.gossamer-threads.com/lists/linuxha/users/79213 , I'm monitoring access time on the SBD partition on the three clusters: average time to execute the dump command is 30ms, sometimes it spikes over 100ms a couple of times in an hour. There's no slow rise from the average when the problem comes, though, here's what it looked like the last time, frequency of the dump command is 2 seconds:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>...<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>real 0m0.031s<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>real 0m0.031s<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>real 0m0.030s<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>real 0m0.030s<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>real 0m0.030s<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>real 0m0.030s<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>real 0m0.031s </span><span lang=EN-US style='font-size:12.0pt;font-family:Wingdings'>ß</span><span lang=EN-US style='font-size:12.0pt'>-- last record on the file, no more logging, server will reboot after the timeout watchdog period<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>...<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>Right before the last cluster reboot I was monitoring Oracle I/O towards its datafiles, to verify whether Oracle could access its partition, on the same LUN as the SBD one, when the SBD countdown start, to identify if it’s an SBD-only problem or a LUN access problem), and there was no sign of Oracle I/O problems during the countdown, it seems Oracle stopped interacting with the I/O monitor software the very moment the Oracle servers rebooted (all servers involved have a common time-server, but I can’t be 100% sure they were in sync when I checked).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>I'm in close contact with the SAN department, the problem might well be the servers losing access to the LUN for some fibre channel matter they still can't see in their SAN logs, but I'd like to be 100% certain the cluster configuration is good. Here are my SBD related questions:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>- is the 1 MB size for the SBD partition strictly mandatory ? in SLES 11 Sp1 HA documentation it's written: "In an environment where all nodes have access to shared storage, a small partition (1MB) is formated for the use with SBD", while here http://linux-ha.org/wiki/SBD_Fencing there is no size suggested for it. At Os setup the SLES partitioner didn't allow us to create a 1MB partition, being it too small, the smallest size available was 7.8MB: can this difference in size introduce the random problem we're experiencing ? <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'>- I've read here http://www.gossamer-threads.com/lists/linuxha/pacemaker/84951 Mr. Lars Marowsky-Bree says: "The new SBD versions will not become stuck on IO anymore". Is the SBD version I'm using one that can become stuck on IO ? I've checked without luck for SLES HA packages newer than the one I'm using, but the SBD being stuck on IO really seems something that would apply to my case.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt'>Thanks and best regards.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt'><o:p> </o:p></span></p></div></body></html>