[ofa-general] OFED vs IB_GOLD IB_SRP Performance results

[MAHMOUD HANAFI]

All,

I have been evaluating srp performance using a DDN IB attached storage. I have looked at both OFED 
and IBGOLD. I have discovered some interesting results. Although the write performance is consistent 
between OFED and IBGOLD, the read performance is not. I looked at various tuning setting but have 
been unable to improve read performance of OFED. I am interested in getting feed back in regards to 
these results.

As you can see in this chart that the IBGD out performs OFED at the larger record Lengths.
READ CHART: http://www.clusteringsolutions.com/openib/Read.png

WRITE CHART: http://www.clusteringsolutions.com/openib/Write.png


Test Setup:
Test software: xdd using direct IO
Sever: Dell 2950 4 core 8GB memory
Storage: DDN S2A 9500
LUN: 4 - 1 per tier (8+1) Blocksize = 4096
IB: SDR cisco
Fiber Channel = 4 Gb/sec Qlogic using qla2400 driver
kernel tested:  2.6.9-42.0.10.ELsmp and 2.6.9-42.0.10.EL_lustre.1.4.10smp
IB Stack: OFED1.1, OFED1.2, and IB_GOLD1.8.3

IBGOLD Setup:
/sys/module/ib_srp/dlid_conf = 0
/sys/module/ib_srp/fmr_cache = 0
/sys/module/ib_srp/ib_ports_mask = -1
/sys/module/ib_srp/max_cmds_per_lun = 1
/sys/module/ib_srp/max_luns = 256
/sys/module/ib_srp/max_srp_targets = 16
/sys/module/ib_srp/max_xfer_sectors_per_io = 8192
/sys/module/ib_srp/refcnt = 16
/sys/module/ib_srp/service_str = <NULL>
/sys/module/ib_srp/srp_discovery_timeout = 60
/sys/module/ib_srp/srp_tracelevel = 2
/sys/module/ib_srp/target_bindings = <NULL>

/sys/block/sdc/queue/max_hw_sectors_kb = 4096
/sys/block/sdc/queue/max_sectors_kb = 4096
/sys/block/sdc/queue/nr_requests = 8192
/sys/block/sdc/queue/read_ahead_kb = 128

OFED Setup:
/sys/module/ib_srp/mellanox_workarounds = 1
/sys/module/ib_srp/refcnt = 11
/sys/module/ib_srp/srp_sg_tablesize = 256
/sys/module/ib_srp/topspin_workarounds = 1

/sys/block/sdd/queue/max_sectors_kb = 4096
/sys/block/sdd/queue/nr_requests = 8192
/sys/block/sdd/queue/read_ahead_kb = 128

Thanks,
-- 
Mahmoud Hanafi
Senior System Administrator
ASC/MSRC
www.asc.hpc.mil
2435 5th Street
WPAFB, OHIO 45433
(937) 255-1536