[ofa-general] ***SPAM*** IBGD/OFED configuration/performance question
dave at thedillows.org
Tue Jan 1 19:26:16 PST 2008
On Tue, 2008-01-01 at 15:28 -0800, Old Apple wrote:
> With OFED 18.104.22.168, everything connects without any
> problem, all 6 disks are seen inside the server. By
> setting max_sect to 16384 and srp_sg_tablesize to 256
> we however obtain only 110 MB/sec per LUN which is too
> low for our DDN system. (To make sure we are not
> simply piping everything into one SDR port of DDN, we
> tried to use only 3 LUNS together, and still never
> went over 110 MB/sec per LUN).
As a quick hack/workaround, add max_cmds_per_lun=5 to the string you are
echoing to add-target under OFED. That should get you better
performance, as the SRP implementation will send requests to the array
when it has no credits available to do so.
For a better fix that will let you queue commands to the DDN for a
single lun when the others are not busy, I sent some patches a weeks ago
to respect the credit handling, and to force a maximum queue length.
Search for subjects:
IB/srp: respect target credit limit
IB/srp: use scatter gather chaining
IB/srp: allow user to control host queue length
The one for scatter gather chaining may not work for pre 2.6.24-rc1
kernels, I haven't checked. It isn't absolutely critical.
With the third one, "...queue length", you can replace the
max_cmds_per_lun=5 with "queue_len=31" to prevent some performance
degradation at full tilt -- I'm still tracking it down, but it wasn't as
bad as seeing 110MB/s, IIRC. You may want to keep the max_cmds_per_lun,
though, if your workload is such that all 6 LUNs will be equally busy.
With these, I can max out a port on the DDN with a single LUN
More information about the general