[ofa-general] Re: Performance penalty of OFED 1.1 versus IBGD 1.8.2

[Michael S. Tsirkin]

> Quoting Roland Fehrenbacher <rf at q-leap.de>:
> Subject: Re: Performance penalty of OFED 1.1 versus IBGD 1.8.2
> 
> >>>>> "MST" == Michael S Tsirkin <mst at mellanox.co.il> writes:
> 
>     >> Quoting Roland Fehrenbacher <rf at q-leap.de>: Subject: Re:
>     >> Performance penalty of OFED 1.1 versus IBGD 1.8.2
> 
>     >> I'm migrating from IBGD 1.8.2 (kernel 2.6.15.7) to OFED 1.1,
>     >> and saw some unpleasant performance drops when using OFED 1.1
>     >> (kernel 2.6.20.1 with included IB drivers). The main drop is in
>     >> throughput as measured by the OSU MPI bandwidth
>     >> benchmark. However, the latency for large packet sizes is also
>     >> worse (see results below). I tried with and without "options
>     >> ib_mthca msi_x=1" (using IBGD, disabling msi_x makes a
>     >> siginficant performance difference of approx. 10%). The IB card
>     >> is a Mellanox MHGS18-XT (PCIe/DDR Firmware 1.2.0) running on an
>     >> Opteron with nForce4 2200 Professional chipset.
>     >>
>     >> Does anybody have an explanation or even better a solution
>     >> to this issue?
> 
>     MST> Could be a BIOS bug. Try setting tune_pci=1. If this helps,
>     MST> contact your BIOS vendor: here's an explanation about what
>     MST> this parameter does:
> 
>     MST> http://www.mail-archive.com/openib-general@openib.org/msg25305.html
> 
>     >> I tried this with no effect. Just to make sure the settings are
>     >> in effect, is there a way I can check this after booting?
> 
>     MST> cat /sys/modules/ib_mthca/parameters/tune_pci
> 
> Ok, the settings are active, but have zero effect. Anything else I
> could check?

No idea. Could be an MPI issue?

-- 
MST