[ofa-general] IO Size more than 48K

[Batwara, Ashish]

Yes. We are using CATC. However, our customers are using Finisar too, but none of the analyzer vendor has IB DDR analyzer and we are looking for one. 
Talked to CATC (Lecroy) 2 weeks ago, and they have this as part of their roadmap and tentative date might be Q3'08. Does anyone know any vendor who is having DDR analyzer currently?

Thanks
Ashish

-----Original Message-----
From: chas williams - CONTRACTOR [mailto:chas at cmf.nrl.navy.mil] 
Sent: Monday, December 03, 2007 11:14 AM
To: koen.segers at VRT.BE
Cc: Batwara, Ashish; Kevin Harms; openib-general at openib.org
Subject: Re: [ofa-general] IO Size more than 48K 

lecroy catc.  its the only 4x analyzer on the market to the best
of my knowledge.

In message <1196690201.6758.10.camel at koenVRT>,Koen Segers writes:
>Can you give more information on which analyzer you used?
>
>
>Regards,
>
>Koen
>
>
>On Fri, 2007-11-30 at 10:48 -0700, Batwara, Ashish wrote:
>> This is what I did as suggested by Vu and it seems to be working.
>> However, when I send 2MB IO, it gets broken into 512K+1MB+512K by SRP
>> as seen on analyzer. I am just wondering what the logic is? On the
>> other side, when we increase the srp_sg_tablesize beyond 256, we are
>> seeing following message in /var/log/messages “Nov 29 21:17:50 p50
>> kernel:   REJ reason 0x3” which indicates “IB_CM_REJ_NO_RESOURCES”, so
>> not sure how to get around to this problem to send larger IO than 1MB
>> in one shot.
>> 
>>  
>> 
>>  
>> 
>> modprobe ib_srp srp_sg_tablesize=256
>> 
>> echo
>> id_ext=200600A0B81138C9,max_sect=4096,ioc_guid=00a0b81112da0003,dgid=fe80000
>00000000000a0b81112da0001,pkey=ffff,service_id=200600a0b81138c9> /sys/class/in
>finiband_srp/srp-mthca0-1/add_target
>> 
>>  
>> 
>> -----Original Message-----
>> From: chas williams - CONTRACTOR [mailto:chas at cmf.nrl.navy.mil] 
>> Sent: Friday, November 30, 2007 11:43 AM
>> To: Kevin Harms
>> Cc: Vu Pham; openib-general at openib.org; Batwara, Ashish
>> Subject: Re: [ofa-general] IO Size more than 48K 
>> 
>>  
>> 
>> addtionally, you might need to echo 'blocks' >
>> 
>> /sys/block/<device/queue/max_hw_segments to increase the size of the
>> 
>> rdma segments.
>> 
>>  
>> 
>> max_hw_segments doesnt exist on all kernels i think.
>> 
>>  
>> 
>> In message <3A453CF1-5FFC-44BF-8F72-7E3EF5AA6E41 at alcf.anl.gov>,Kevin
>> Harms writ
>> 
>> es:
>> 
>> > 
>> 
>> >     you may also have to go to /sys/block/sdX/queue and echo 1024
>> >  
>> 
>> >max_sectors_kb
>> 
>> >     if you use the srp_daemon you can also add:
>> 
>> >     a max_sect=2048 to /etc/srp_daemon.conf
>> 
>> > 
>> 
>> >kevin
>> 
>> > 
>> 
>> >On Nov 29, 2007, at 11:08 AM, Vu Pham wrote:
>> 
>> > 
>> 
>> >> 
>> 
>> >>> Hi,
>> 
>> >>> We are using OFED-1.2, and using xdd and some other tools, and  
>> 
>> >>> trying to
>> 
>> >>> send 1/2MB IOs, but what we are seeing in analyzer traces, that  
>> 
>> >>> memory
>> 
>> >>> descriptor in SRP command shows max. 48K which means 1MB I/Os
>> has  
>> 
>> >>> broken
>> 
>> >>> into smaller SRP request from initiator.
>> 
>> >>> How can I have this I/O directly going to target? What parameter
>> I  
>> 
>> >>> need
>> 
>> >>> to change?
>> 
>> >>> 
>> 
>> >>> 
>> 
>> >> 
>> 
>> >> module param srp_sg_tablesize (default is 12 ie. 12 x 4K = 48K)
>> 
>> >> and/or
>> 
>> >> max_sect=yyy in echo id_ext=xxx,...,max_sect=1024,service_id=
>> > /sys/ 
>> 
>> >> class/infiniband_srp/...
>> 
>> >> 
>> 
>> >> -vu
>> 
>> >> 
>> 
>> >>> Thanks
>> 
>> >>> Ashish
>> 
>> >>> _______________________________________________
>> 
>> >>> general mailing list
>> 
>> >>> general at lists.openfabrics.org
>> 
>> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>> 
>> >>> 
>> 
>> >>> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-gene
>> 
>> >ral
>> 
>> >>> 
>> 
>> >> 
>> 
>> >> _______________________________________________
>> 
>> >> general mailing list
>> 
>> >> general at lists.openfabrics.org
>> 
>> >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>> 
>> >> 
>> 
>> >> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-gener
>> 
>> >al
>> 
>> >> 
>> 
>> > 
>> 
>> >_______________________________________________
>> 
>> >general mailing list
>> 
>> >general at lists.openfabrics.org
>> 
>> >http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>> 
>> > 
>> 
>> >To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-general
>> 
>> > 
>> 
>> 
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>> 
>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-gener
>al
>*** Disclaimer ***
>
>Vlaamse Radio- en Televisieomroep
>Auguste Reyerslaan 52, 1043 Brussel
>
>nv van publiek recht
>BTW BE 0244.142.664
>RPR Brussel
>http://www.vrt.be/disclaimer
>
>