[ofa-general] Need OFED1.1 ib_srp max_hw_sectors_kb help!

[MAHMOUD HANAFI]

Some test data for OFED1.1. I am going to run OFED1.2 and IBGOLD.

* 1 Lun per Tier(8+1) with block size of 4096KB
* Using xdd writing to 4 luns.
* Using 1 IB host port to 1 IB DDN Port.

The 700MB/sec appear to be a host limit. Because using 2 IB Host ports and 2
DDN IB ports IO still max out at 700MB/sec

Note that write level off at 512KB Request Size. I did verified IO Request Lengths on DDN

= WRITE DirectIO==
IO
Size  Throughput
KB      MB/sec
---  ---------
16     46.711
32     93.293
64    185.098
128   348.522
256   547.118
512   671.227
1024  697.149
2048  680.645
4096  692.067
8192  710.564

= READ DirectIO==
IO
Size  Throughput
KB      MB/sec
---  ---------
16     54.856
32    104.526
64    191.592
128   312.586
256   462.460
512   471.877
1024  509.806
2048  535.050
4096  543.130
8192  565.176

Martin W. Schlining III wrote:
> Wonder why it halves the value?  I'll have to try that myself.
> 
> If you are using OFED 1.2, you can also load the ib_srp module with an
> option to increase the size of the scatter gather lists. The default size is
> 12 which is way too small. I don't think this option exists for OFED 1.1. In
> 1.1, you have to modify ib_srp.h and recompile the module ib_srp.o.
> 
> modprobe ib_srp srp_sg_tablesize=256
> 
> Fiber channel drivers also set the max_sect field in their drivers to 65535
> (0xffff) to eliminate any restrictions. Perhaps the same value for SRP will
> help?
> 
> - Martin
> 
> -----Original Message-----
> From: MAHMOUD HANAFI [mailto:hanafim.ctr at asc.hpc.mil] 
> Sent: Friday, June 01, 2007 2:56 PM
> To: Martin W. Schlining III
> Cc: 'MAHMOUD HANAFI'
> Subject: Re: [ofa-general] Need OFED1.1 ib_srp max_hw_sectors_kb help!
> 
> I didn't get a answer on the email list but I figured it out.
> You can pass "max_sect=xxx" option during initialization of the srp traget.
> If you have upgraded to
> OFED1.2 you can set a line in /etc/srp_daemon.conf "A max_sect=2096" (you
> will need the "A") and then run the srp_daemon.sh. Only thing I have noticed
> is what ever you set in the max_sect value it always takes 1/2 of the value.
> 
> 
> 
> -Mahmoud
> 
> Martin W. Schlining III wrote:
>>  Hello,
>>
>> Did you ever get an answer to this? I'm a bit curious myself what the 
>> response was.
>>
>> Regards,
>> Martin Schlining
>> Senior Software Engineer
>> mschlining at datadirectnet.com
>>
>> -----Original Message-----
>> From: general-bounces at lists.openfabrics.org
>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of MAHMOUD 
>> HANAFI
>> Sent: Tuesday, May 29, 2007 6:58 PM
>> To: general at lists.openfabrics.org
>> Subject: [ofa-general] Need OFED1.1 ib_srp max_hw_sectors_kb help!
>>
>> All,
>>
>> I am using OFED1.1 with CISCO HCA/switch and DDN Storage. I am able to 
>> load and perform IO to the DDN via srp driver. But, the 
>> max_hw_sectors_kb for the device is getting set to 64kb. Any one else 
>> seen this issue? Same host and storage with fiber channel doesn't have 
>> this problem. It set max_hw_sectors_kb correctly to 4096KB.
>>
>> Thanks,
>> --
>> Mahmoud Hanafi
>> Senior System Administrator
>> ASC/MSRC
>> www.asc.hpc.mil
>> 2435 5th Street
>> WPAFB, OHIO 45433
>> (937) 255-1536
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-general
>>
>>
> 
> --
> Mahmoud Hanafi
> Senior System Administrator
> ASC/MSRC
> www.asc.hpc.mil
> 2435 5th Street
> WPAFB, OHIO 45433
> (937) 255-1536
> 
> 

-- 
Mahmoud Hanafi
Senior System Administrator
ASC/MSRC
www.asc.hpc.mil
2435 5th Street
WPAFB, OHIO 45433
(937) 255-1536