<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='color:#1F497D'>We are using 2 IBM FAStT900’s.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Normally the timestamps of the
messages on both the SFS and the IB host match.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Thanks<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>jeroen<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Scott Weitzenkamp
(sweitzen) [mailto:sweitzen@cisco.com] <br>
<b>Sent:</b> woensdag 19 december 2007 18:32<br>
<b>To:</b> Jeroen Van Aken; general@lists.openfabrics.org<br>
<b>Subject:</b> RE: [ofa-general] ***SPAM*** SFS 3012 SRP problem<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>If you have a Cisco supoport contract, you should open a case with
the Cisco TAC.</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>What kind of FC storage are you using?</span><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>The chassis syslog message show the host is unresponsive (the
OUT_SERVICE and IN_SERVICE message). Do the timing of these messages
match the ib_srp messages on the host?</span><span style='font-size:12.0pt;
font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<div>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
</div>
<p><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>Scott
Weitzenkamp<br>
SQA and Release Manager<br>
Server Virtualization Business Unit<br>
Cisco Systems<o:p></o:p></span></p>
<div>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
</div>
<blockquote style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt;
margin-left:3.75pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p>
<div class=MsoNormal align=center style='text-align:center'><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'>
<hr size=2 width="100%" align=center>
</span></div>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> general-bounces@lists.openfabrics.org
[mailto:general-bounces@lists.openfabrics.org] <b>On Behalf Of </b>Jeroen Van
Aken<br>
<b>Sent:</b> Wednesday, December 19, 2007 6:54 AM<br>
<b>To:</b> general@lists.openfabrics.org<br>
<b>Subject:</b> [ofa-general] ***SPAM*** SFS 3012 SRP problem</span><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<p class=MsoNormal>Hello<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>We are doing some SRP tests with the Cisco SFS 3012 Gateway.
We connected 4 hosts, each with 2 infiniband cables on one dual infiniband card
to the SFS3012 gateway. The gateway is also connected to our fibre channel
storage. The ofed used is OFED-1.3-beta2 on each of the hosts. The
infiniband cards used are InfiniBand: Mellanox Technologies MT25208 InfiniHost
III Ex (rev a0) and Mellanox Technologies MT23108 InfiniHost (rev a1)
cards.<o:p></o:p></p>
<p class=MsoNormal>When generating heavy load over the switch (by reading from
our FC storage over all the luns simultaneously), we sometimes get the
following errors:<o:p></o:p></p>
<p class=MsoNormal>On the hosts: <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Dec 13 13:07:54 gpfs4n1 syslog-ng[8212]: STATS: dropped 0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 13:20:26 gpfs4n1 run_srp_daemon[8422]: failed
srp_daemon: [HCA=mthca0] [port=1] [exit status=110]. Will try to restart
srp_daemon periodically. No mor<o:p></o:p></p>
<p class=MsoNormal>e warnings will be issued in the next 7200 seconds if the
same problem repeats<o:p></o:p></p>
<p class=MsoNormal>Dec 13 13:20:27 gpfs4n1 run_srp_daemon[8428]: starting
srp_daemon: [HCA=mthca0] [port=1]<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:01:20 gpfs4n1 sshd[8539]: Accepted
keyboard-interactive/pam for root from 172.16.0.18 port 3545 ssh2<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:07:55 gpfs4n1 syslog-ng[8212]: STATS: dropped 0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 syslog-ng[8212]: Changing
permissions on special file /dev/xconsole<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 syslog-ng[8212]: Changing
permissions on special file /dev/tty10<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed send status
12<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed send status
12<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>On the switch ts_log<o:p></o:p></p>
<p class=MsoNormal>**************************************SWITCH
LOG***************************************************************<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:04:30 topspin-cc ib_sm.x[1357]: [INFO]:
Configuration caused by multicast membership change<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:05:49 topspin-cc ib_sm.x[1383]: [INFO]: Session
not initiated: Cold Sync Limit exceeded for Standby SM guid
00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:07:49 topspin-cc ib_sm.x[1383]: [INFO]: Initialize
a backup session with Standby SM guid 00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:07:59 topspin-cc ib_sm.x[1383]: [INFO]: Session
initialization failed with Standby SM guid 00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:09:59 topspin-cc ib_sm.x[1383]: [INFO]: Initialize
a backup session with Standby SM guid 00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:10:09 topspin-cc ib_sm.x[1383]: [INFO]: Session
initialization failed with Standby SM guid 00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:06 topspin-cc ib_sm.x[1357]: [INFO]: Generate
SM OUT_OF_SERVICE trap for GID=fe:80:00:00:00:00:00:00:00:05:ad:00:00:1d:ce:21<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:06 topspin-cc ib_sm.x[1357]: [INFO]: Generate
SM OUT_OF_SERVICE trap for GID=fe:80:00:00:00:00:00:00:00:05:ad:00:00:1d:ce:22<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:06 topspin-cc ib_sm.x[1357]: [INFO]: Configuration
caused by discovering removed ports<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:07 topspin-cc ib_sm.x[1357]: [INFO]:
Configuration caused by multicast membership change<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:09 topspin-cc ib_sm.x[1383]: [INFO]: Session
not initiated: Cold Sync Limit exceeded for Standby SM guid
00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:18 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for read, err=11, t1=1, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:22 topspin-cc last message repeated 4 times<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:36 topspin-cc ib_sm.x[1357]: [INFO]: Configuration
caused by discovering new ports<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:37 topspin-cc ib_sm.x[1357]: [INFO]: Generate
SM IN_SERVICE trap for GID=fe:80:00:00:00:00:00:00:00:05:ad:00:00:1d:ce:21<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:37 topspin-cc ib_sm.x[1357]: [INFO]: Generate
SM IN_SERVICE trap for GID=fe:80:00:00:00:00:00:00:00:05:ad:00:00:1d:ce:22<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:38 topspin-cc ib_sm.x[1357]: [INFO]:
Configuration caused by multicast membership change<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:28 topspin-cc chassis_mgr.x[1084]: [WARN]:
tsIpcMessageSend failed, fd=28, vp=2, err=104, Connection reset by peer<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:39 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:39 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:40 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:46 topspin-cc web_agent.x[1370]: [INFO]: ipc:
select(fd=3) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:50 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:50 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:50 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:18 topspin-cc ib_sm.x[1383]: [INFO]: Session
not initiated: Cold Sync Limit exceeded for Standby SM guid
00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:38 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:38 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:40 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:49 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:50 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:50 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:15:00 topspin-cc web_agent.x[1370]: [INFO]: ipc:
select(fd=3) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:15:00 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:15:00 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:15:00 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>It looks like some of the log entries are incomplete.<o:p></o:p></p>
<p class=MsoNormal>I think it is a switch related issue: first of all because
of the strange format of the logs, and second because when this error occurs in
the switch, no SRP communication is possible on either of the IB hosts. I already
tried increasing the Node timeout, and set RENICE_IB_MAD to yes as described in
this thread: <a
href="http://lists.openfabrics.org/pipermail/general/2007-May/036465.html">http://lists.openfabrics.org/pipermail/general/2007-May/036465.html</a>.
But this didn’t help.<o:p></o:p></p>
<p class=MsoNormal>This issue occurs randomly. So it isn’t easily
reproduced.<o:p></o:p></p>
<p class=MsoNormal>Does anybody have an idea what went wrong?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Thanks in advance!<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Jeroen Van Aken<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
</blockquote>
</div>
</div>
</body>
</html>