[ofa-general] IO Size more than 48K

[Batwara, Ashish]

This is what I did as suggested by Vu and it seems to be working.
However, when I send 2MB IO, it gets broken into 512K+1MB+512K by SRP as
seen on analyzer. I am just wondering what the logic is? On the other
side, when we increase the srp_sg_tablesize beyond 256, we are seeing
following message in /var/log/messages "Nov 29 21:17:50 p50 kernel:
REJ reason 0x3" which indicates "IB_CM_REJ_NO_RESOURCES", so not sure
how to get around to this problem to send larger IO than 1MB in one
shot.

 

 

modprobe ib_srp srp_sg_tablesize=256

echo
id_ext=200600A0B81138C9,max_sect=4096,ioc_guid=00a0b81112da0003,dgid=fe8
000000000000000a0b81112da0001,pkey=ffff,service_id=200600a0b81138c9>
/sys/class/infiniband_srp/srp-mthca0-1/add_target

 

-----Original Message-----
From: chas williams - CONTRACTOR [mailto:chas at cmf.nrl.navy.mil] 
Sent: Friday, November 30, 2007 11:43 AM
To: Kevin Harms
Cc: Vu Pham; openib-general at openib.org; Batwara, Ashish
Subject: Re: [ofa-general] IO Size more than 48K 

 

addtionally, you might need to echo 'blocks' >

/sys/block/<device/queue/max_hw_segments to increase the size of the

rdma segments.

 

max_hw_segments doesnt exist on all kernels i think.

 

In message <3A453CF1-5FFC-44BF-8F72-7E3EF5AA6E41 at alcf.anl.gov>,Kevin
Harms writ

es:

> 

>     you may also have to go to /sys/block/sdX/queue and echo 1024 >  

>max_sectors_kb

>     if you use the srp_daemon you can also add:

>     a max_sect=2048 to /etc/srp_daemon.conf

> 

>kevin

> 

>On Nov 29, 2007, at 11:08 AM, Vu Pham wrote:

> 

>> 

>>> Hi,

>>> We are using OFED-1.2, and using xdd and some other tools, and  

>>> trying to

>>> send 1/2MB IOs, but what we are seeing in analyzer traces, that  

>>> memory

>>> descriptor in SRP command shows max. 48K which means 1MB I/Os has  

>>> broken

>>> into smaller SRP request from initiator.

>>> How can I have this I/O directly going to target? What parameter I  

>>> need

>>> to change?

>>> 

>>> 

>> 

>> module param srp_sg_tablesize (default is 12 ie. 12 x 4K = 48K)

>> and/or

>> max_sect=yyy in echo id_ext=xxx,...,max_sect=1024,service_id= > /sys/


>> class/infiniband_srp/...

>> 

>> -vu

>> 

>>> Thanks

>>> Ashish

>>> _______________________________________________

>>> general mailing list

>>> general at lists.openfabrics.org

>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

>>> 

>>> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-gene

>ral

>>> 

>> 

>> _______________________________________________

>> general mailing list

>> general at lists.openfabrics.org

>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

>> 

>> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-gener

>al

>> 

> 

>_______________________________________________

>general mailing list

>general at lists.openfabrics.org

>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

> 

>To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general

> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20071130/6a96ec0d/attachment.html>