[ofa-general] Performance penalty of OFED 1.1 versus IBGD 1.8.2

[Pavel Shamis (Pasha)]

Roland Fehrenbacher wrote:
>>>>>> "Pavel" == Pavel Shamis <(Pasha)" <pasha at dev.mellanox.co.il>> writes:
> 
>     Pavel> Hi Roland,
>     >> >> I'm migrating from IBGD 1.8.2 (kernel 2.6.15.7) to OFED 1.1,
>     >> >> and saw some unpleasant performance drops when using OFED
>     >> 1.1 >> (kernel 2.6.20.1 with included IB drivers). The main
>     >> drop is in >> throughput as measured by the OSU MPI bandwidth
>     >> >> benchmark. However, the latency for large packet sizes is
>     >> also >> worse (see results below). I tried with and without
>     >> "options >> ib_mthca msi_x=1" (using IBGD, disabling msi_x
>     >> makes a >> siginficant performance difference of
>     >> approx. 10%). The IB card >> is a Mellanox MHGS18-XT (PCIe/DDR
>     >> Firmware 1.2.0) running on an >> Opteron with nForce4 2200
>     >> Professional chipset.
>     >> >> 
>     >> >> Does anybody have an explanation or even better a solution
>     >> to >> this issue?
>     >> 
> 
>     Pavel> Please try to add follow mvapich parameter :
>     Pavel> VIADEV_DEFAULT_MTU=MTU2048
>     >> Thanks for the suggestion. Unfortunately, it didn't improve the
>     >> simple bandwidth results. Bi-directional bandwidth increased by
>     >> 3% though. Any more ideas?
> 
>     Pavel> 3% is good start :-) Please also try to add this one:
>     Pavel> VIADEV_MAX_RDMA_SIZE=4194304
> 
> This brought another 2% in bi-directional bandwidth, but still nothing
> in uni-directional bandwidth.
> 
> mvapich version is 0.9.8
0.9.8 was not distributed (and tested) with OFED 1.1 :-(
Please try to use package distributed with OFED 1.1 version.

Pasha.