[ofa-general] OFED vs IB_GOLD IB_SRP Performance results

[Vu Pham]

Hi MAHMOUD,


> All,
> 
> I have been evaluating srp performance using a DDN IB attached storage. 
> I have looked at both OFED and IBGOLD. I have discovered some 
> interesting results. Although the write performance is consistent 
> between OFED and IBGOLD, the read performance is not. I looked at 
> various tuning setting but have been unable to improve read performance 
> of OFED. I am interested in getting feed back in regards to these results.
> 
> As you can see in this chart that the IBGD out performs OFED at the 
> larger record Lengths.
> READ CHART: http://www.clusteringsolutions.com/openib/Read.png
> 
> WRITE CHART: http://www.clusteringsolutions.com/openib/Write.png
> 
> 
> Test Setup:
> Test software: xdd using direct IO
> Sever: Dell 2950 4 core 8GB memory
> Storage: DDN S2A 9500
> LUN: 4 - 1 per tier (8+1) Blocksize = 4096
> IB: SDR cisco
> Fiber Channel = 4 Gb/sec Qlogic using qla2400 driver
> kernel tested:  2.6.9-42.0.10.ELsmp and 2.6.9-42.0.10.EL_lustre.1.4.10smp
> IB Stack: OFED1.1, OFED1.2, and IB_GOLD1.8.3
> 
> IBGOLD Setup:
> /sys/module/ib_srp/dlid_conf = 0
> /sys/module/ib_srp/fmr_cache = 0
> /sys/module/ib_srp/ib_ports_mask = -1
> /sys/module/ib_srp/max_cmds_per_lun = 1
> /sys/module/ib_srp/max_luns = 256
> /sys/module/ib_srp/max_srp_targets = 16
> /sys/module/ib_srp/max_xfer_sectors_per_io = 8192
> /sys/module/ib_srp/refcnt = 16
> /sys/module/ib_srp/service_str = <NULL>
> /sys/module/ib_srp/srp_discovery_timeout = 60
> /sys/module/ib_srp/srp_tracelevel = 2
> /sys/module/ib_srp/target_bindings = <NULL>
> 
> /sys/block/sdc/queue/max_hw_sectors_kb = 4096
> /sys/block/sdc/queue/max_sectors_kb = 4096
> /sys/block/sdc/queue/nr_requests = 8192
> /sys/block/sdc/queue/read_ahead_kb = 128
> 
> OFED Setup:
> /sys/module/ib_srp/mellanox_workarounds = 1
> /sys/module/ib_srp/refcnt = 11
> /sys/module/ib_srp/srp_sg_tablesize = 256
> /sys/module/ib_srp/topspin_workarounds = 1
> 
> /sys/block/sdd/queue/max_sectors_kb = 4096


For OFED drivers:
what is the max_cmd_per_lun? (default is = SRP_SQ_SIZE = 63)
You can set max_cmd_per_lun when adding target - please try 
1, 2, 4, 8...

You can check *cat /sys/class/scsi_host/hostXXX/cmd_per_lun*

Another tuning requires edit/recompile srp driver
+ vi ib_srp.h and change SRP_RQ_SHIFT to 7 --> this will 
increase .can_queue and send_wq/recv_wq to 128 --> this can 
be translate to the increase of queue_depth
+ recompile srp driver

-vu