[ofa-general] OFED vs IB_GOLD IB_SRP Performance results

[MAHMOUD HANAFI]

The max_hw_sectors_kb was set at 4096KB. On the DDN I verified that the request sizes where correct.
I haven tried larger max_hw_sectors_kb it had no effect. Setting the cmd_per_lun=1 improved reads 
slightly but not much.

chas williams - CONTRACTOR wrote:
> In message <4665A1FA.1000506 at asc.hpc.mil>,MAHMOUD HANAFI writes:
>> OFED Setup:
>> /sys/module/ib_srp/mellanox_workarounds = 1
>> /sys/module/ib_srp/refcnt = 11
>> /sys/module/ib_srp/srp_sg_tablesize = 256
>> /sys/module/ib_srp/topspin_workarounds = 1
>>
>> /sys/block/sdd/queue/max_sectors_kb = 4096
>> /sys/block/sdd/queue/nr_requests = 8192
>> /sys/block/sdd/queue/read_ahead_kb = 128
> 
> what is the max_hw_sectors_kb for the ofed target?  unless you specified
> max_sect= during login, i suspect you are getting the system defaults.
> typically this is 512 sectors i think, which is where your performance
> seems to start to diverge.
> 

-- 
Mahmoud Hanafi
Senior System Administrator
ASC/MSRC
www.asc.hpc.mil
2435 5th Street
WPAFB, OHIO 45433
(937) 255-1536