[openib-general] Performance Degradation with OFED v. Voltaire(lustre)

[Or Gerlitz]

Bernadat, Philippe wrote:
> I checked. We apparently never go through this path (with lustre) 

Philippe,

Lustre's openib nld (o2ibnld) always go through

rdma_resolve_route --> cma_resolve_ib_route --> cma_query_ib_route

!!!

please add a sanity check printk to the __init code of the rdma_cm 
module @ drivers/infiniband/core/cma.c to see that the code you are 
working on is actually loaded into the kernel

But this will not help you, the Voltaire SM/SA that you are using will 
not return you 1K MTU based on the fixed cma-tavor-quirk patch that 
Michael has sent. This is actually correct also for the Open SM/SA when 
it does not apply a tavor quirk of its own...

So basically, for the time being please patch Lustre o2ibnal to set the 
MTU to 1K (either always or under some mod param whose default is true), 
till the issue is discussed and decided over this list.

Per the best of my knowledge (Mellanox people please correct me if i am 
wrong): basically if you use 2K MTU for IB RC with MLX/Tavor you get 50% 
BW drop, and if you use 2K MTU for IB/RC with MLX/Arble or Sinai you get 
5% BW increase. And the BW drop problem holds if either of the parties 
is tavor.

Thanks for pointing on the problem and raising the issue!

Or.