<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.2180" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006>Hi Amit,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006>A manual firmware upgrade fixed the problem. I was
incorrectly expecting that the ibed installer would have automatically flashed
the correct firmware.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006>The code is now still being compiled with PCI_EX but
the only impact is that the vbuf size(sort of packet size) calculation
in mvapich becomes incorrect(mvapich calculates vbuf size based on the
weather its a PCI_X/PCI_EX cards). </SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006>I believe we would need to get MVAPICH changed so
that when it compiles for PCI_X it compiles it with SRQ support and the
mvapich.make script needs to be fixed so that it correctly identifies the IB
card as PCI_X. </SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006>-Amit</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=723445215-05052006></SPAN></FONT> </DIV><FONT face=Arial
color=#0000ff size=2></FONT><FONT face=Arial color=#0000ff size=2></FONT><FONT
face=Arial color=#0000ff size=2></FONT><FONT face=Arial color=#0000ff
size=2></FONT><FONT face=Arial color=#0000ff size=2></FONT><FONT face=Arial
color=#0000ff size=2></FONT><FONT face=Arial color=#0000ff size=2></FONT><FONT
face=Arial color=#0000ff size=2></FONT><FONT face=Arial color=#0000ff
size=2></FONT><FONT face=Arial color=#0000ff size=2></FONT><FONT face=Arial
color=#0000ff size=2></FONT><FONT face=Arial color=#0000ff size=2></FONT><FONT
face=Arial color=#0000ff size=2></FONT><FONT face=Arial color=#0000ff
size=2></FONT><FONT face=Arial color=#0000ff size=2></FONT><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Amit Krig [mailto:amitk@mellanox.co.il]
<BR><B>Sent:</B> Thursday, May 04, 2006 5:48 PM<BR><B>To:</B> Amit Mehrotra
(amehrotr); Openfabrics-ewg@openib.org<BR><B>Subject:</B> RE: [openfabrics-ewg]
MVAPICH on PCI-X fails with [0] Abort: Couldn'tmodify SRQ
limit<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><SPAN class=546560312-04052006><FONT face="Courier New" color=#0000ff
size=2>Hi Amit,</FONT></SPAN></DIV>
<DIV><SPAN class=546560312-04052006><FONT size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=546560312-04052006><FONT size=2>Please note that SRQ
limit event is being supported from fw version 3.4.0
<P></FONT></SPAN><SPAN class=546560312-04052006><FONT size=2>Can
you specify the </FONT></SPAN><SPAN class=546560312-04052006><FONT
size=2>FW version that you use, SRQ size, number of outstanding WR in
the SRQ when the modify SRQ command was executed, requested SRQ limit value and
QP transport type.</FONT></SPAN></P>
<P><SPAN class=546560312-04052006><FONT size=2></FONT></SPAN> </P>
<P><SPAN class=546560312-04052006><FONT size=2>Amit</FONT></SPAN></P></DIV>
<DIV><SPAN class=546560312-04052006><FONT size=2><FONT face="Courier New"
color=#0000ff></FONT> </DIV></FONT></SPAN><FONT size=2></FONT><FONT
size=2></FONT><FONT size=2></FONT><FONT size=2></FONT><FONT size=2></FONT><FONT
size=2></FONT><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> openfabrics-ewg-bounces@openib.org
[mailto:openfabrics-ewg-bounces@openib.org] <B>On Behalf Of </B>Amit Mehrotra
(amehrotr)<BR><B>Sent:</B> Tuesday, May 02, 2006 2:36 PM<BR><B>To:</B>
Openfabrics-ewg@openib.org<BR><B>Subject:</B> [openfabrics-ewg] MVAPICH on PCI-X
fails with [0] Abort: Couldn'tmodify SRQ limit<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>Configuration:-
RHEL4U3,ia32,rc3,PCI-X</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>I have been seeing
the following error when I try to run MVAPICH test programs</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2>----</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>[0] Abort: Couldn't
modify SRQ limit<BR> at line 999 in file
viainit.c<BR>-----</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>On
debugging the issue it seems that MVAPICH is being incorrectly
compiled for the PCI_EX cards rather than PCI_X cards. From the MPI
code it seems that PCI-X cards do not support modifications of SRQs. The
source of the problem lies in a bug in the mvapich.make script(new IBED
addition) which incorrectly always returns the card as PCI_EX . I have
appended a diff with the fix. I am not sure as to how the patch can
be correctly generated as the whole MVAPICH is in the form of a
zipped tarball.</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>There seems to be
one more issue in the script where it deviates from the MVAPICH build. The
script is treating the older PCI_EX cards(cards with the lspci signature of
15b3:6278) as PCI-X. Was this done because these cards also don't support
resizing SRQs? </FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2>-Amit</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2>-------------------</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>diff -u
mvapich.make.old mvapich.make<BR>--- mvapich.make.old
2006-05-02 15:32:11.000000000 +0530<BR>+++
mvapich.make 2006-05-02
15:33:37.000000000 +0530<BR>@@ -251,13 +251,13
@@<BR>
DEF_BUILDID="$DEF_BUILDID"<BR>
fi<BR> fi<BR>-if (/sbin/lspci -n | grep "15b3:6282" | wc -l | tr -d
'[:space:]'); then<BR>+if (test `/sbin/lspci -n | grep "15b3:6282" | wc -l | tr
-d '[:space:]'` -gt 0); then<BR> #
Arbel<BR> CFLAGS="$CFLAGS
-D_PCI_EX_"<BR>-elif (/sbin/lspci -n | grep -E "15b3:5e8c|15b3:6274" | wc -l |
tr -d '[:space:]'); then<BR>+elif (test `/sbin/lspci -n | grep -E
"15b3:5e8c|15b3:6274" | wc -l | tr -d '[:space:]'` -gt 0);
then<BR> #
Sinai<BR> CFLAGS="$CFLAGS
-D_PCI_EX_"<BR>-elif (/sbin/lspci -n | grep -E "15b3:5a44|15b3:6278" | wc -l |
tr -d '[:space:]'); then<BR>+elif (test `/sbin/lspci -n | grep -E
"15b3:5a44|15b3:6278" | wc -l | tr -d '[:space:]'` -gt 0);
then<BR> #
Tavor<BR> CFLAGS="$CFLAGS
-D_PCI_X_"<BR> fi<BR></FONT></SPAN><SPAN class=133162410-02052006><FONT
face=Arial size=2>-----------</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2> </DIV></FONT></SPAN></BODY></HTML>