[ofa-general] Performance penalty of OFED 1.1 versus IBGD 1.8.2

[Roland Fehrenbacher]

>>>>> "Pavel" == Pavel Shamis <(Pasha)" <pasha at dev.mellanox.co.il>> writes:

    Pavel> Hi Roland,
    >> I'm migrating from IBGD 1.8.2 (kernel 2.6.15.7) to OFED 1.1,
    >> and saw some unpleasant performance drops when using OFED 1.1
    >> (kernel 2.6.20.1 with included IB drivers). The main drop is in
    >> throughput as measured by the OSU MPI bandwidth
    >> benchmark. However, the latency for large packet sizes is also
    >> worse (see results below). I tried with and without "options
    >> ib_mthca msi_x=1" (using IBGD, disabling msi_x makes a
    >> siginficant performance difference of approx. 10%). The IB card
    >> is a Mellanox MHGS18-XT (PCIe/DDR Firmware 1.2.0) running on an
    >> Opteron with nForce4 2200 Professional chipset.
    >> 
    >> Does anybody have an explanation or even better a solution to
    >> this issue?

    Pavel> Please try to add follow mvapich parameter :
    Pavel> VIADEV_DEFAULT_MTU=MTU2048

Thanks for the suggestion. Unfortunately, it didn't improve the simple
bandwidth results. Bi-directional bandwidth increased by 3%
though. Any more ideas?

Roland

> ------------------------------------------------------------------------
> 
> IBGD
> --------
> 
> # OSU MPI Bandwidth Test (Version 2.1)
> # Size          Bandwidth (MB/s)
> 1               0.830306
> 2               1.642710
> 4               3.307494
> 8               6.546477
> 16              13.161954
> 32              26.395154
> 64              52.913060
> 128             101.890547
> 256             172.227478
> 512             383.296292
> 1024            611.172247
> 2048            830.147571
> 4096            1068.057366
> 8192            1221.262520
> 16384           1271.771983
> 32768           1369.702828
> 65536           1426.124683
> 131072          1453.781151
> 262144          1457.297992
> 524288          1464.625860
> 1048576         1468.953875
> 2097152         1470.614903
> 4194304         1471.607758
> 
> # OSU MPI Latency Test (Version 2.1)
> # Size          Latency (us)
> 0               3.03
> 1               3.03
> 2               3.04
> 4               3.03
> 8               3.03
> 16              3.04
> 32              3.11
> 64              3.23
> 128             3.49
> 256             3.83
> 512             4.88
> 1024            6.31
> 2048            8.60
> 4096            11.02
> 8192            15.78
> 16384           28.85
> 32768           39.82
> 65536           60.30
> 131072          106.65
> 262144          196.47
> 524288          374.62
> 1048576         730.79
> 2097152         1442.32
> 4194304         2864.80
> 
> OFED 1.1
> ---------
> 
> # OSU MPI Bandwidth Test (Version 2.2)
> # Size          Bandwidth (MB/s)
> 1               0.698614
> 2               1.463192
> 4               2.941852
> 8               5.859464
> 16              11.697510
> 32              23.339031
> 64              46.403081
> 128             92.013928
> 256             182.918388
> 512             315.076923
> 1024            500.083937
> 2048            765.294564
> 4096            1003.652513
> 8192            1147.640312
> 16384           1115.803139
> 32768           1221.120298
> 65536           1282.328447
> 131072          1315.715608
> 262144          1331.456393
> 524288          1340.691793
> 1048576         1345.650404
> 2097152         1349.279211
> 4194304         1350.489883
> 
> # OSU MPI Latency Test (Version 2.2)
> # Size          Latency (us)
> 0               2.99
> 1               3.03
> 2               3.06
> 4               3.03
> 8               3.03
> 16              3.04
> 32              3.12
> 64              3.27
> 128             3.96
> 256             4.29
> 512             4.99
> 1024            6.53
> 2048            9.08
> 4096            11.92
> 8192            17.39
> 16384           31.05
> 32768           43.47
> 65536           67.17
> 131072          115.30
> 262144          212.33
> 524288          405.20
> 1048576         790.45
> 2097152         1558.88
> 4194304         3095.17
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general