[ofa-general] Performance penalty of OFED 1.1 versus IBGD 1.8.2

[Sayantan Sur]

Hi,

* On Feb,3 Sayantan Sur<surs at cse.ohio-state.edu> wrote :
> Hi Roland,
> 
> * On Feb,2 Pavel Shamis (Pasha)<pasha at dev.mellanox.co.il> wrote :
> > Roland Fehrenbacher wrote:
> > >>>>>>"Pavel" == Pavel Shamis <(Pasha)" <pasha at dev.mellanox.co.il>> writes:
> > >
> > >    Pavel> Hi Roland,
> > >    >> >> I'm migrating from IBGD 1.8.2 (kernel 2.6.15.7) to OFED 1.1,
> > >    >> >> and saw some unpleasant performance drops when using OFED
> > >    >> 1.1 >> (kernel 2.6.20.1 with included IB drivers). The main
> > >    >> drop is in >> throughput as measured by the OSU MPI bandwidth
> > >    >> >> benchmark. However, the latency for large packet sizes is
> > >    >> also >> worse (see results below). I tried with and without
> > >    >> "options >> ib_mthca msi_x=1" (using IBGD, disabling msi_x
> > >    >> makes a >> siginficant performance difference of
> > >    >> approx. 10%). The IB card >> is a Mellanox MHGS18-XT (PCIe/DDR
> > >    >> Firmware 1.2.0) running on an >> Opteron with nForce4 2200
> > >    >> Professional chipset.
> > >    >> >> 
> > >    >> >> Does anybody have an explanation or even better a solution
> > >    >> to >> this issue?
> > >    >> 
> > >
> > >    Pavel> Please try to add follow mvapich parameter :
> > >    Pavel> VIADEV_DEFAULT_MTU=MTU2048
> > >    >> Thanks for the suggestion. Unfortunately, it didn't improve the
> > >    >> simple bandwidth results. Bi-directional bandwidth increased by
> > >    >> 3% though. Any more ideas?
> > >
> > >    Pavel> 3% is good start :-) Please also try to add this one:
> > >    Pavel> VIADEV_MAX_RDMA_SIZE=4194304
> > >
> > >This brought another 2% in bi-directional bandwidth, but still nothing
> > >in uni-directional bandwidth.
> > >
> > >mvapich version is 0.9.8
> > 0.9.8 was not distributed (and tested) with OFED 1.1 :-(
> > Please try to use package distributed with OFED 1.1 version.
> 
> MVAPICH-0.9.8 was tested by the MVAPICH team on OFED 1.1. It is being
> used at several production clusters with OFED 1.1.
> 
> I ran the bandwidth test on our Opteron nodes, AMD Processor 254 (2.8
> GHz), with Mellanox dual-port DDR cards. I can see a peak bandwidth of
> 1402 MillionBytes/sec as reported by OSU Bandwidth test. On the same
> machines, I ran ib_rdma_bw (in the perftest module of OFED-1.1), which
> reports lower Gen2 level performance numbers. The peak bw reported by
> ib_rdma_bw is 1307.09 MegaBytes/sec (=1338.09*1.048 = 1402
> MillionBytes/sec). So, the lower level numbers match up to what is
> reported by MPI.

The above was done with OFED-1.1. Using IBGD-1.8.2 on the same machines
and saw 1402 MillionBytes/sec peak bandwidth. This is the same as
reported by OFED-1.1.

> I'm wondering how your lower-level ib_rdma_bw numbers look like? Are
> they matching up with what OSU BW test reports? If they are, then it is
> likely some other issue than MPI.
> 
> We also have a MVAPICH-0.9.9 beta version out. You could give that a try
> too, if you want. We will be making the full release soon.

In addition, you can check the following URL w.r.t. performance
numbers.

http://nowlab.cse.ohio-state.edu/projects/mpi-iba/performance/mvapich/opteron/MVAPICH-0.9.8-opteron-gen2-DDR.html

Thanks,
Sayantan.

> 
> Thanks,
> Sayantan.
> 
> -- 
> http://www.cse.ohio-state.edu/~surs
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

-- 
http://www.cse.ohio-state.edu/~surs