[openib-general] Performance Degradation with OFED v. Voltaire

[Tziporet Koren]

Bernadat, Philippe wrote:
>> Have you tried running with
>>
>> options ib_mthca tune_pci =1
>>
>>     
>
> My understanding is that this is not required anymore with OFED-1.1 - It
> used to make a siginifciant differences with OFED-1.0, but I didn't
> observe it with OFED-1.1
>
> And again, the user mode performance if comparable between VIB and OFED.
>
> Philippe
>
>   
Its not related to OFED 1.1 or OFED 1.0, but to difference between OFED 
and VAPI.
In VAPI this setting was always done. In OFED we do not do it by default 
and you need this parameter.

See this note on mthca release notes:

4. Performance degradation due to wrong BIOS configuration:
   The PCI Express spec. requires BIOS to set the MaxReadReq register
   for each card for maximum performance and stability. 

   If you are seeing bandwidth performance degradation, you can try forcing
   the card to behave out of PCI Express spec. by setting the tune_pci=1 module
   parameter.  This tune_pci=1 option was the default setting in OFED
   1.0, which might have masked performance degradation on some systems.

   If tune_pci=1 improves bandwidth, please report the issue to your 
   BIOS vendor. Please note that Mellanox Technologies does not recommend using
   tune_pci=1 in production systems: working with tune_pci=1 option set is
   untested and is known to trigger stability issues on some platforms.


Can you please try it.

Tziporet