[ofa-general] Performance penalty of OFED 1.1 versus IBGD 1.8.2

[Pavel Shamis (Pasha)]

>     Pavel> Hi Roland,
>     >> I'm migrating from IBGD 1.8.2 (kernel 2.6.15.7) to OFED 1.1,
>     >> and saw some unpleasant performance drops when using OFED 1.1
>     >> (kernel 2.6.20.1 with included IB drivers). The main drop is in
>     >> throughput as measured by the OSU MPI bandwidth
>     >> benchmark. However, the latency for large packet sizes is also
>     >> worse (see results below). I tried with and without "options
>     >> ib_mthca msi_x=1" (using IBGD, disabling msi_x makes a
>     >> siginficant performance difference of approx. 10%). The IB card
>     >> is a Mellanox MHGS18-XT (PCIe/DDR Firmware 1.2.0) running on an
>     >> Opteron with nForce4 2200 Professional chipset.
>     >> 
>     >> Does anybody have an explanation or even better a solution to
>     >> this issue?
> 
>     Pavel> Please try to add follow mvapich parameter :
>     Pavel> VIADEV_DEFAULT_MTU=MTU2048
> 
> Thanks for the suggestion. Unfortunately, it didn't improve the simple
> bandwidth results. Bi-directional bandwidth increased by 3%
> though. Any more ideas?
3% is good start :-)
Please also try to add this one:
VIADEV_MAX_RDMA_SIZE=4194304

-Pasha

> 
> Roland
> 
>> ------------------------------------------------------------------------
>>
>> IBGD
>> --------
>>
>> # OSU MPI Bandwidth Test (Version 2.1)
>> # Size          Bandwidth (MB/s)
>> 1               0.830306
>> 2               1.642710
>> 4               3.307494
>> 8               6.546477
>> 16              13.161954
>> 32              26.395154
>> 64              52.913060
>> 128             101.890547
>> 256             172.227478
>> 512             383.296292
>> 1024            611.172247
>> 2048            830.147571
>> 4096            1068.057366
>> 8192            1221.262520
>> 16384           1271.771983
>> 32768           1369.702828
>> 65536           1426.124683
>> 131072          1453.781151
>> 262144          1457.297992
>> 524288          1464.625860
>> 1048576         1468.953875
>> 2097152         1470.614903
>> 4194304         1471.607758
>>
>> # OSU MPI Latency Test (Version 2.1)
>> # Size          Latency (us)
>> 0               3.03
>> 1               3.03
>> 2               3.04
>> 4               3.03
>> 8               3.03
>> 16              3.04
>> 32              3.11
>> 64              3.23
>> 128             3.49
>> 256             3.83
>> 512             4.88
>> 1024            6.31
>> 2048            8.60
>> 4096            11.02
>> 8192            15.78
>> 16384           28.85
>> 32768           39.82
>> 65536           60.30
>> 131072          106.65
>> 262144          196.47
>> 524288          374.62
>> 1048576         730.79
>> 2097152         1442.32
>> 4194304         2864.80
>>
>> OFED 1.1
>> ---------
>>
>> # OSU MPI Bandwidth Test (Version 2.2)
>> # Size          Bandwidth (MB/s)
>> 1               0.698614
>> 2               1.463192
>> 4               2.941852
>> 8               5.859464
>> 16              11.697510
>> 32              23.339031
>> 64              46.403081
>> 128             92.013928
>> 256             182.918388
>> 512             315.076923
>> 1024            500.083937
>> 2048            765.294564
>> 4096            1003.652513
>> 8192            1147.640312
>> 16384           1115.803139
>> 32768           1221.120298
>> 65536           1282.328447
>> 131072          1315.715608
>> 262144          1331.456393
>> 524288          1340.691793
>> 1048576         1345.650404
>> 2097152         1349.279211
>> 4194304         1350.489883
>>
>> # OSU MPI Latency Test (Version 2.2)
>> # Size          Latency (us)
>> 0               2.99
>> 1               3.03
>> 2               3.06
>> 4               3.03
>> 8               3.03
>> 16              3.04
>> 32              3.12
>> 64              3.27
>> 128             3.96
>> 256             4.29
>> 512             4.99
>> 1024            6.53
>> 2048            9.08
>> 4096            11.92
>> 8192            17.39
>> 16384           31.05
>> 32768           43.47
>> 65536           67.17
>> 131072          115.30
>> 262144          212.33
>> 524288          405.20
>> 1048576         790.45
>> 2097152         1558.88
>> 4194304         3095.17
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general