[Openib-windows] SRP tuning

[Fabian Tillier]

Hi Yossi,

On 10/1/06, Yossi Leybovich <sleybo at mellanox.co.il> wrote:
>
> Fab
> One of the people in the list ask about tuning option to the SRP driver
>
>
> "...Now in regards to performance - In the linux environment, on the
> host/initiator there were numerous parameters that could be toyed with to
> boost performance.  Specifically, there is a parameter called
> max_xfer_sectors_per_io in a file called modprobe-openib.conf.  By changing
> the value of this parameter, performance via srp to srpt and finally to
> disks could go up and down dramatically..."
>
> Do you have nay idea of similar parameter in windows SRP driver ?
> I found  p_config->MaximumTransferLength       =
> SCSI_MAXIMUM_TRANSFER_SIZE;
> which is 1K*1K.
>
> Does this value can be change ? Do you think it will affect the performance
> ?

We should probably leave this value uninitialized.  It is initialized
by the port driver to SP_UNINITIALIZED_VALUE to indicate unlimited
maximum transfer size.

The real limit to the size of transfers is the NumberOfPhysicalBreaks
and SrbExtensionSize.  The SRB extension is used to hold the SRP
requests, including the S/G list for direct and indirect I/O requests.
 The size of the SRB extension limits the number of S/G entries, and
thus the NumberOfPhysicalBreaks.

Right now, the SRB extension is large enough to hold a SRP IU of 340
bytes, which is enough to send up to 17 S/G entries in the partial
memory descriptor list.  Note that this S/G list is referenced by the
indirect table memory descriptor, so even if the initiator to target
IU size doesn't support the 17 S/G entries to be put in the SRP
command, the 17 S/G entry limit still holds.

17 S/G entries is the maximum needed to transfer 64KB in a single I/O.

A target that defines a larger initiator to target IU will have larger
SRB extensions, and thus support a larger transfer size.

It would be intersting to see how the S/G limit affects bandwidth.
Increasing SRP_MAX_IU_SIZE will achieve this.  You won't run into the
SCSI_MAX_TRANSFER_SIZE limit until you have 257 S/G entries, which
means a SRP_MAX_IU_SIZE of 4180 bytes (from it's current 340 bytes).

- Fab