[ofa-general] Performance penalty of OFED 1.1 versus IBGD 1.8.2

[Roland Fehrenbacher]

>>>>> "Pavel" == Pavel Shamis <(Pasha)" <pasha at dev.mellanox.co.il>> writes:

    Pavel> Hi Roland,
    >> >> I'm migrating from IBGD 1.8.2 (kernel 2.6.15.7) to OFED 1.1,
    >> >> and saw some unpleasant performance drops when using OFED
    >> 1.1 >> (kernel 2.6.20.1 with included IB drivers). The main
    >> drop is in >> throughput as measured by the OSU MPI bandwidth
    >> >> benchmark. However, the latency for large packet sizes is
    >> also >> worse (see results below). I tried with and without
    >> "options >> ib_mthca msi_x=1" (using IBGD, disabling msi_x
    >> makes a >> siginficant performance difference of
    >> approx. 10%). The IB card >> is a Mellanox MHGS18-XT (PCIe/DDR
    >> Firmware 1.2.0) running on an >> Opteron with nForce4 2200
    >> Professional chipset.
    >> >> 
    >> >> Does anybody have an explanation or even better a solution
    >> to >> this issue?
    >> 

    Pavel> Please try to add follow mvapich parameter :
    Pavel> VIADEV_DEFAULT_MTU=MTU2048
    >> Thanks for the suggestion. Unfortunately, it didn't improve the
    >> simple bandwidth results. Bi-directional bandwidth increased by
    >> 3% though. Any more ideas?

    Pavel> 3% is good start :-) Please also try to add this one:
    Pavel> VIADEV_MAX_RDMA_SIZE=4194304

This brought another 2% in bi-directional bandwidth, but still nothing
in uni-directional bandwidth.

mvapich version is 0.9.8

Roland