[ofa-general] ***SPAM*** IBGD/OFED configuration/performance question

[Dave Dillow]

On Tue, 2008-01-01 at 15:28 -0800, Old Apple wrote:
> With OFED 1.2.5.4, everything connects without any
> problem, all 6 disks are seen inside the server. By
> setting max_sect to 16384 and srp_sg_tablesize to 256
> we however obtain only 110 MB/sec per LUN which is too
> low for our DDN system. (To make sure we are not
> simply piping everything into one SDR port of DDN, we
> tried to use only 3 LUNS together, and still never
> went over 110 MB/sec per LUN). 

As a quick hack/workaround, add max_cmds_per_lun=5 to the string you are
echoing to add-target under OFED. That should get you better
performance, as the SRP implementation will send requests to the array
when it has no credits available to do so.

For a better fix that will let you queue commands to the DDN for a
single lun when the others are not busy, I sent some patches a weeks ago
to respect the credit handling, and to force a maximum queue length.
Search for subjects:
IB/srp: respect target credit limit
IB/srp: use scatter gather chaining
IB/srp: allow user to control host queue length

The one for scatter gather chaining may not work for pre 2.6.24-rc1
kernels, I haven't checked. It isn't absolutely critical.

With the third one, "...queue length", you can replace the
max_cmds_per_lun=5 with "queue_len=31" to prevent some performance
degradation at full tilt -- I'm still tracking it down, but it wasn't as
bad as seeing 110MB/s, IIRC. You may want to keep the max_cmds_per_lun,
though,  if your workload is such that all 6 LUNs will be equally busy.

With these, I can max out a port on the DDN with a single LUN
(~750MB/s).