[ofa-general] IO Size more than 48K

[Koen Segers]

Can you give more information on which analyzer you used?


Regards,

Koen


On Fri, 2007-11-30 at 10:48 -0700, Batwara, Ashish wrote:
> This is what I did as suggested by Vu and it seems to be working.
> However, when I send 2MB IO, it gets broken into 512K+1MB+512K by SRP
> as seen on analyzer. I am just wondering what the logic is? On the
> other side, when we increase the srp_sg_tablesize beyond 256, we are
> seeing following message in /var/log/messages “Nov 29 21:17:50 p50
> kernel:   REJ reason 0x3” which indicates “IB_CM_REJ_NO_RESOURCES”, so
> not sure how to get around to this problem to send larger IO than 1MB
> in one shot.
> 
>  
> 
>  
> 
> modprobe ib_srp srp_sg_tablesize=256
> 
> echo
> id_ext=200600A0B81138C9,max_sect=4096,ioc_guid=00a0b81112da0003,dgid=fe8000000000000000a0b81112da0001,pkey=ffff,service_id=200600a0b81138c9> /sys/class/infiniband_srp/srp-mthca0-1/add_target
> 
>  
> 
> -----Original Message-----
> From: chas williams - CONTRACTOR [mailto:chas at cmf.nrl.navy.mil] 
> Sent: Friday, November 30, 2007 11:43 AM
> To: Kevin Harms
> Cc: Vu Pham; openib-general at openib.org; Batwara, Ashish
> Subject: Re: [ofa-general] IO Size more than 48K 
> 
>  
> 
> addtionally, you might need to echo 'blocks' >
> 
> /sys/block/<device/queue/max_hw_segments to increase the size of the
> 
> rdma segments.
> 
>  
> 
> max_hw_segments doesnt exist on all kernels i think.
> 
>  
> 
> In message <3A453CF1-5FFC-44BF-8F72-7E3EF5AA6E41 at alcf.anl.gov>,Kevin
> Harms writ
> 
> es:
> 
> > 
> 
> >     you may also have to go to /sys/block/sdX/queue and echo 1024
> >  
> 
> >max_sectors_kb
> 
> >     if you use the srp_daemon you can also add:
> 
> >     a max_sect=2048 to /etc/srp_daemon.conf
> 
> > 
> 
> >kevin
> 
> > 
> 
> >On Nov 29, 2007, at 11:08 AM, Vu Pham wrote:
> 
> > 
> 
> >> 
> 
> >>> Hi,
> 
> >>> We are using OFED-1.2, and using xdd and some other tools, and  
> 
> >>> trying to
> 
> >>> send 1/2MB IOs, but what we are seeing in analyzer traces, that  
> 
> >>> memory
> 
> >>> descriptor in SRP command shows max. 48K which means 1MB I/Os
> has  
> 
> >>> broken
> 
> >>> into smaller SRP request from initiator.
> 
> >>> How can I have this I/O directly going to target? What parameter
> I  
> 
> >>> need
> 
> >>> to change?
> 
> >>> 
> 
> >>> 
> 
> >> 
> 
> >> module param srp_sg_tablesize (default is 12 ie. 12 x 4K = 48K)
> 
> >> and/or
> 
> >> max_sect=yyy in echo id_ext=xxx,...,max_sect=1024,service_id=
> > /sys/ 
> 
> >> class/infiniband_srp/...
> 
> >> 
> 
> >> -vu
> 
> >> 
> 
> >>> Thanks
> 
> >>> Ashish
> 
> >>> _______________________________________________
> 
> >>> general mailing list
> 
> >>> general at lists.openfabrics.org
> 
> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> >>> 
> 
> >>> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-gene
> 
> >ral
> 
> >>> 
> 
> >> 
> 
> >> _______________________________________________
> 
> >> general mailing list
> 
> >> general at lists.openfabrics.org
> 
> >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> >> 
> 
> >> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-gener
> 
> >al
> 
> >> 
> 
> > 
> 
> >_______________________________________________
> 
> >general mailing list
> 
> >general at lists.openfabrics.org
> 
> >http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> > 
> 
> >To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
> 
> > 
> 
> 
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
*** Disclaimer ***

Vlaamse Radio- en Televisieomroep
Auguste Reyerslaan 52, 1043 Brussel

nv van publiek recht
BTW BE 0244.142.664
RPR Brussel
http://www.vrt.be/disclaimer