<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.2180" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><SPAN class=469355518-02052006><FONT face=Arial
size=2>Amit,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=469355518-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=469355518-02052006><FONT face=Arial size=2>I
haven't had any problems with this config.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=469355518-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><FONT face=Arial size=2>Protest Run Results</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2>Chassis Branch: TopspinOS-2.6.0<BR>Chassis Build:
build171<BR>IB Hosts: svbu-qaclus-1 svbu-qaclus-2<BR>IB Hosts OS:
openib_i686_smp_rhel4_34 openib_i686_smp_rhel4_34<BR>Host Branch:
OFED-1.0<BR>Host Build: buildrc3<BR>Topology:
/data/home/scott/cluster_switch.main/qa/tests/smoke/Lk3.topo<BR>Logs:
/data/home/scott/builds/TopspinOS-2.6.0/build171/protest/Lk3/050206_115207<BR>Logs
URL: <A
href="http://svbu-borg.cisco.com/~releng/data/home/scott/builds/TopspinOS-2">http://svbu-borg.cisco.com/~releng/data/home/scott/builds/TopspinOS-2</A>.<BR>6.0/build171/protest/Lk3/050206_115207</FONT></DIV>
<DIV> </DIV><FONT face=Arial size=2>
<DIV><BR>Scenario file: MpiOsuBenchmarks.scen</DIV>
<DIV> </DIV>
<DIV>###TEST-D: MPI home dir is
/usr/local/ibed/mpi/gcc/mvapich-0.9.7-mlx2.1.0.</DIV>
<DIV> </DIV>
<DIV>###TEST-D: MVAPICH OpenIB osu_latency test on hosts svbu-qaclus-1
svbu-qaclus-2<BR># OSU MPI Latency Test (Version 2.1)<BR>#
Size Latency
(us)<BR>0
5.16<BR>1
5.27<BR>2
5.32<BR>4
5.27<BR>8
5.37<BR>16
5.38<BR>32
5.49<BR>64
5.55<BR>128
6.70<BR>256
7.35<BR>512
8.35<BR>1024
10.33<BR>2048
14.51<BR>4096
19.64<BR>8192
59.40<BR>16384
72.00<BR>32768
98.60<BR>65536
152.97<BR>131072
259.36<BR>262144
472.56<BR>524288
895.95<BR>1048576
1748.12<BR>2097152
3476.54<BR>4194304
8703.27<BR>###TEST-D: MVAPICH OpenIB osu_bw test on hosts svbu-qaclus-1
svbu-qaclus-2<BR># OSU MPI Bandwidth Test (Version 2.1)<BR>#
Size Bandwidth
(MB/s)<BR>1
0.115860<BR>2
0.234141<BR>4
0.468693<BR>8
0.938176<BR>16
1.801834<BR>32
3.774280<BR>64
7.574665<BR>128
15.227615<BR>256
30.406622<BR>512
62.068835<BR>1024
126.576020<BR>2048
223.283704<BR>4096
358.732809<BR>8192
285.252288<BR>16384
388.095564<BR>32768
474.323906<BR>65536
531.422345<BR>131072
562.801189<BR>262144
579.782217<BR>524288
588.706779<BR>1048576
593.214942<BR>2097152
595.447328<BR>4194304
594.820726<BR>###TEST-D: MVAPICH OpenIB osu_bibw test on hosts svbu-qaclus-1
svbu-qaclus-2<BR># OSU MPI Bidirectional Bandwidth Test (Version 2.1)<BR>#
Size Bi-Bandwidth
(MB/s)<BR>1
0.143480<BR>2
0.291655<BR>4
0.579100<BR>8
1.132267<BR>16
2.310026<BR>32
4.616616<BR>64
9.075500<BR>128
19.643435<BR>256
39.267568<BR>512
73.597916<BR>1024
158.550363<BR>2048
290.699402<BR>4096
509.814370<BR>8192
326.919911<BR>16384
459.685017<BR>32768
571.193714<BR>65536
652.835363<BR>131072
700.064301<BR>262144
726.029468<BR>524288
739.861749<BR>1048576
747.233197<BR>2097152
751.168876<BR>4194304
698.340369<BR>###TEST-D: MVAPICH OpenIB osu_bcast test on hosts svbu-qaclus-1
svbu-qaclus-2<BR># OSU MPI_Bcast Latency Test (Version 1.0)<BR>#
Size Latency
(us)<BR> 1
5.899<BR> 2
5.884<BR> 4
5.887<BR> 8
5.882<BR> 16 5.987<BR>
32 5.860<BR>
64 5.917<BR>
128 7.202<BR>
256 7.981<BR>
512 9.371<BR> 1024
11.292<BR> 2048
15.053<BR> 4096
20.031<BR> 8192
63.100<BR>16384 75.669<BR>###TEST-D: MVAPICH OpenIB
mpi_multibw test on hosts svbu-qaclus-1 svbu-qaclus-2<BR># PathScale Modified
OSU MPI Bandwidth Test (Version 2.2)<BR># Running on 1 procs per node<BR>#
Size Aggregate Bandwidth
(MB/s)
Messages/s<BR>1
0.116658
116658.464118<BR>2
0.234265
117132.451180<BR>4
0.471091
117772.625225<BR>8
0.933336
116666.970487<BR>16
1.808325
113020.290674<BR>32
3.776229
118007.154184<BR>64
7.559009
118109.509661<BR>128
15.161947
118452.711457<BR>256
30.493207
119114.088963<BR>512
62.340430
121758.651523<BR>1024
126.154498
123197.751641<BR>2048
225.834353
110270.680060<BR>4096
362.919482
88603.389080<BR>8192
285.539695
34855.919788<BR>16384
391.398443
23889.065154<BR>32768
474.238097
14472.598171<BR>65536
531.765959
8114.104596<BR>131072
562.829510
4294.048382<BR>262144
580.052829
2212.725940<BR>524288
588.813185
1123.072023<BR>1048576
593.293871
565.809127<BR>2097152
595.487749
283.950686<BR>4194304
594.906214
141.836694<BR>InfiniBand.Performance.MPI.OsuBenchmarks.MVAPICH.OpenIB -->
PASS<BR></FONT></DIV>
<DIV dir=ltr align=left><SPAN class=469355518-02052006><FONT
face=Courier></FONT></SPAN> </DIV>
<DIV> </DIV>
<DIV align=left><FONT face=Arial size=2>Scott Weitzenkamp</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>SQA and Release Manager</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Server Virtualization Business
Unit</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Cisco Systems</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> openfabrics-ewg-bounces@openib.org
[mailto:openfabrics-ewg-bounces@openib.org] <B>On Behalf Of </B>Amit Mehrotra
(amehrotr)<BR><B>Sent:</B> Tuesday, May 02, 2006 4:36 AM<BR><B>To:</B>
Openfabrics-ewg@openib.org<BR><B>Subject:</B> [openfabrics-ewg] MVAPICH on
PCI-X fails with [0] Abort: Couldn'tmodify SRQ limit<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>Configuration:-
RHEL4U3,ia32,rc3,PCI-X</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>I have been seeing
the following error when I try to run MVAPICH test
programs</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2>----</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>[0] Abort:
Couldn't modify SRQ limit<BR> at line 999 in file
viainit.c<BR>-----</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>On
debugging the issue it seems that MVAPICH is being incorrectly
compiled for the PCI_EX cards rather than PCI_X cards. From the MPI
code it seems that PCI-X cards do not support modifications of SRQs. The
source of the problem lies in a bug in the mvapich.make script(new IBED
addition) which incorrectly always returns the card as PCI_EX . I have
appended a diff with the fix. I am not sure as to how the patch can
be correctly generated as the whole MVAPICH is in the form of
a zipped tarball.</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>There seems to be
one more issue in the script where it deviates from the MVAPICH build. The
script is treating the older PCI_EX cards(cards with the lspci signature
of 15b3:6278) as PCI-X. Was this done because these cards also don't
support resizing SRQs? </FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2>-Amit</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2>-------------------</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial size=2>diff -u
mvapich.make.old mvapich.make<BR>--- mvapich.make.old
2006-05-02 15:32:11.000000000 +0530<BR>+++
mvapich.make 2006-05-02
15:33:37.000000000 +0530<BR>@@ -251,13 +251,13
@@<BR>
DEF_BUILDID="$DEF_BUILDID"<BR>
fi<BR> fi<BR>-if (/sbin/lspci -n | grep "15b3:6282" | wc -l | tr -d
'[:space:]'); then<BR>+if (test `/sbin/lspci -n | grep "15b3:6282" | wc -l |
tr -d '[:space:]'` -gt 0); then<BR>
# Arbel<BR> CFLAGS="$CFLAGS
-D_PCI_EX_"<BR>-elif (/sbin/lspci -n | grep -E "15b3:5e8c|15b3:6274" | wc -l |
tr -d '[:space:]'); then<BR>+elif (test `/sbin/lspci -n | grep -E
"15b3:5e8c|15b3:6274" | wc -l | tr -d '[:space:]'` -gt 0);
then<BR> #
Sinai<BR> CFLAGS="$CFLAGS
-D_PCI_EX_"<BR>-elif (/sbin/lspci -n | grep -E "15b3:5a44|15b3:6278" | wc -l |
tr -d '[:space:]'); then<BR>+elif (test `/sbin/lspci -n | grep -E
"15b3:5a44|15b3:6278" | wc -l | tr -d '[:space:]'` -gt 0);
then<BR> #
Tavor<BR> CFLAGS="$CFLAGS
-D_PCI_X_"<BR> fi<BR></FONT></SPAN><SPAN class=133162410-02052006><FONT
face=Arial size=2>-----------</FONT></SPAN></DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=133162410-02052006><FONT face=Arial
size=2> </DIV></BLOCKQUOTE></FONT></SPAN></BODY></HTML>