<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>Hello<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>We are doing some SRP tests with the Cisco SFS 3012 Gateway.
We connected 4 hosts, each with 2 infiniband cables on one dual infiniband card
to the SFS3012 gateway. The gateway is also connected to our fibre channel
storage. The ofed used is OFED-1.3-beta2 on each of the hosts. The
infiniband cards used are InfiniBand: Mellanox Technologies MT25208 InfiniHost
III Ex (rev a0) and Mellanox Technologies MT23108 InfiniHost (rev a1)
cards.<o:p></o:p></p>
<p class=MsoNormal>When generating heavy load over the switch (by reading from
our FC storage over all the luns simultaneously), we sometimes get the
following errors:<o:p></o:p></p>
<p class=MsoNormal>On the hosts: <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Dec 13 13:07:54 gpfs4n1 syslog-ng[8212]: STATS: dropped 0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 13:20:26 gpfs4n1 run_srp_daemon[8422]: failed
srp_daemon: [HCA=mthca0] [port=1] [exit status=110]. Will try to restart
srp_daemon periodically. No mor<o:p></o:p></p>
<p class=MsoNormal>e warnings will be issued in the next 7200 seconds if the
same problem repeats<o:p></o:p></p>
<p class=MsoNormal>Dec 13 13:20:27 gpfs4n1 run_srp_daemon[8428]: starting
srp_daemon: [HCA=mthca0] [port=1]<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:01:20 gpfs4n1 sshd[8539]: Accepted
keyboard-interactive/pam for root from 172.16.0.18 port 3545 ssh2<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:07:55 gpfs4n1 syslog-ng[8212]: STATS: dropped 0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 syslog-ng[8212]: Changing
permissions on special file /dev/xconsole<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 syslog-ng[8212]: Changing
permissions on special file /dev/tty10<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:01 gpfs4n1 kernel: SRP abort called<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed send status
12<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed send status
12<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:02 gpfs4n1 kernel: ib_srp: failed receive
status 5<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>On the switch ts_log<o:p></o:p></p>
<p class=MsoNormal>**************************************SWITCH
LOG***************************************************************<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:04:30 topspin-cc ib_sm.x[1357]: [INFO]:
Configuration caused by multicast membership change<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:05:49 topspin-cc ib_sm.x[1383]: [INFO]: Session
not initiated: Cold Sync Limit exceeded for Standby SM guid
00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:07:49 topspin-cc ib_sm.x[1383]: [INFO]: Initialize
a backup session with Standby SM guid 00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:07:59 topspin-cc ib_sm.x[1383]: [INFO]: Session
initialization failed with Standby SM guid 00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:09:59 topspin-cc ib_sm.x[1383]: [INFO]: Initialize
a backup session with Standby SM guid 00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:10:09 topspin-cc ib_sm.x[1383]: [INFO]: Session
initialization failed with Standby SM guid 00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:06 topspin-cc ib_sm.x[1357]: [INFO]: Generate
SM OUT_OF_SERVICE trap for GID=fe:80:00:00:00:00:00:00:00:05:ad:00:00:1d:ce:21<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:06 topspin-cc ib_sm.x[1357]: [INFO]: Generate
SM OUT_OF_SERVICE trap for GID=fe:80:00:00:00:00:00:00:00:05:ad:00:00:1d:ce:22<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:06 topspin-cc ib_sm.x[1357]: [INFO]:
Configuration caused by discovering removed ports<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:07 topspin-cc ib_sm.x[1357]: [INFO]:
Configuration caused by multicast membership change<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:09 topspin-cc ib_sm.x[1383]: [INFO]: Session
not initiated: Cold Sync Limit exceeded for Standby SM guid
00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:18 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for read, err=11, t1=1, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:22 topspin-cc last message repeated 4 times<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:36 topspin-cc ib_sm.x[1357]: [INFO]:
Configuration caused by discovering new ports<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:37 topspin-cc ib_sm.x[1357]: [INFO]: Generate
SM IN_SERVICE trap for GID=fe:80:00:00:00:00:00:00:00:05:ad:00:00:1d:ce:21<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:37 topspin-cc ib_sm.x[1357]: [INFO]: Generate
SM IN_SERVICE trap for GID=fe:80:00:00:00:00:00:00:00:05:ad:00:00:1d:ce:22<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:12:38 topspin-cc ib_sm.x[1357]: [INFO]:
Configuration caused by multicast membership change<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:3<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>13 14:12:4<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:28 topspin-cc chassis_mgr.x[1084]: [WARN]:
tsIpcMessageSend failed, fd=28, vp=2, err=104, Connection reset by peer<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:39 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:39 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:40 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:46 topspin-cc web_agent.x[1370]: [INFO]: ipc:
select(fd=3) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:50 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:50 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:13:50 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:18 topspin-cc ib_sm.x[1383]: [INFO]: Session
not initiated: Cold Sync Limit exceeded for Standby SM guid
00:05:ad:00:00:08:94:5d<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:38 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:38 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:40 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:49 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:50 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:14:50 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:15:00 topspin-cc web_agent.x[1370]: [INFO]: ipc:
select(fd=3) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:15:00 topspin-cc chassis_mgr.x[1084]: [INFO]: ipc:
select(fd=28) failed for write, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:15:00 topspin-cc chassis_mgr.x[1084]: [INFO]:
tsIpcMessageSend failed, fd=28, vp=2, err=11, Resource temporarily unavailable<o:p></o:p></p>
<p class=MsoNormal>Dec 13 14:15:00 topspin-cc snmp_agent.x[1208]: [INFO]: ipc:
select(fd=5) failed for read, err=11, t1=10, t2=0<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>It looks like some of the log entries are incomplete.<o:p></o:p></p>
<p class=MsoNormal>I think it is a switch related issue: first of all because
of the strange format of the logs, and second because when this error occurs in
the switch, no SRP communication is possible on either of the IB hosts. I
already tried increasing the Node timeout, and set RENICE_IB_MAD to yes as
described in this thread: <a
href="http://lists.openfabrics.org/pipermail/general/2007-May/036465.html">http://lists.openfabrics.org/pipermail/general/2007-May/036465.html</a>.
But this didn’t help.<o:p></o:p></p>
<p class=MsoNormal>This issue occurs randomly. So it isn’t easily
reproduced.<o:p></o:p></p>
<p class=MsoNormal>Does anybody have an idea what went wrong?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Thanks in advance!<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Jeroen Van Aken<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>