[ofw] WinIB SRP driver and 64k IO's?

[Daniel.Cummins at schange.com]

Tzachi,

You are correct.. there are 2 SG lists.. the IB scatter gather and the 
StorPort scatter gather list. 

The issue we are seeing is with the StorPort scatter gather list size 
which seems to be limited to 16 entries (16 * 4096 page size = 64KB IO).  
The reason we want this size increased is to be able to scale single 
stream bandwidth by feeding the storage backend larger IOs.  Basically 
pusing the latency down to the backend, keeping the disks busy while 
freeing up the host.  This will give us greater bandwidth.

The StorPort scatter gather list size is controlled by setting 
PORT_CONFIGURATION_INFORMATION::NumberOfPhysicalBreaks.  On entry to 
srp_find_adapter() StorPort has defaulted this to 16.  I was told by 
Microsoft that I should be able to set this to 254. (StorPort adds 1). 
When I set this value it does not stick.  A matter of fact NT is not even 
reading the location.  This was confirmed by setting a break on access to 
NumberOfPhysicalBreaks just before return from FindAdapter() and hitting 
GO in WinDbg.

What I did find out is that the number of map registers allocated to the 
adpater is being limited to 16 by an internal call to IoGetDmaAdapter(). 
This function is called by NT during adapter initialization right after we 
return from FindAdapter().  According to Microsoft this is what is capping 
the scatter gather list size.

As it stands right now they have yet to tell me why IoGetDmaAdapter() is 
limiting the number of map registers.  I will let you know once I find out 
more.

Daniel Cummins
SeaChange International
32 Mill Street, Greenville, NH 03048
Phone (603) 878-6414
mailto:daniel.cummins at schange.com

This e-mail and any attachments may contain information which is SeaChange 
International confidential.  The information enclosed is intended only for 
the addressees herein and may not be copied or forwarded without 
permission from SeaChange International.

Hi Steve,

There are two questions that I need to understand:
1) What SG lists are you talking about? The IB ones (post send/recive)?

2) why is 64kb not enough. Does the code send out one request and only
once it is finished it is sending the next request? Assuming you want to
reach to a BW of 1000MB/sec than this means that you should have a
request once every 64us.
Do you know if the time that it takes to handle a request is bigger that
64us?

Thanks
Tzachi 

> -----Original Message-----
> From: ofw-bounces at lists.openfabrics.org 
> [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Steve 
Valimaki
> Sent: Monday, June 25, 2007 10:16 PM
> To: ofw at lists.openfabrics.org
> Subject: [ofw] WinIB SRP driver and 64k IO's?
> 
> Doing some basic testing (iometer, xdd) of the windows WinIB (Mellanox
> 1.3) SRP driver, its seems that I've not been able to get 
> larger the 64K IO's to my DataDirect Networks Storage array. 
> The host is Windows server
> 2003 SP2. I suspect it may be the need to increase the size 
> of the SG list, is there a known limitation?
> 
> Thanks,
> Steve
> 
> --
> Steve Valimaki
> Field Applications Engineer
> DataDirect Networks Federal LLC
> Cell Phone:   (443) 415-4947
> Office Phone: (410) 309-9300 x152
> 
> http://www.datadirectnet.com
> 
> DDN Support Announcement mailing list:
> http://www.ddnsupport.com/announce
> 
> 
> 
> _______________________________________________
> ofw mailing list
> ofw at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofw/attachments/20070626/86295a4e/attachment.html>