[openib-general] Srp question

[Makia Minich]

On 8/31/06 3:34 PM, "Roland Dreier" <rdreier at cisco.com> wrote:

>     Makia> We are attempting to do some performance testing of the SRP driver
> (with a
>     Makia> DDN target) and are seeing some poor results:
> 
>     Makia> ~120MB/s per lun with 1 sgp_dd
>     Makia> ~80MB/s per lun with 4 sgp_dd
> 
>     Makia> Previously we had attempted the same tests with IBGold and got the
>     Makia> following:
> 
>     Makia> ~150MB/s per lun with 1 sgp_dd
>     Makia> ~600MB/s per lun with 4 sgp_dd
> 
> Were these tests with the same kernels otherwise?  If not, there may
> be unrelated changes to the SCSI stack that affect synthetic
> benchmarks like this.  (I seem to remember a change in the not too
> distant patch that affected the largest IO it is possible to submit
> through the SG interface).

The kernels in question were 2.6.9.22.0.2 and 2.6.9-34.EL.  I'll have to
find some changelogs to see if there were changes to the SCSI stack.

>     Makia> To achieve the results in IBGold, we were able to set the
>     Makia> srp module option "max_xfer_sectors_per_io=4096", but can't
>     Makia> seem to find an equivalent option in the OFED SRP drivers.
> 
> When connecting to the target (the echo to the add_target file), you
> can add ",max_sect=4096" to the string you pass in.

We did add this (pardon my typo of max_sects below).  I did find that by
appending the max_sect= at the end of the line we were seeing strange
behaviour (it seemed that the parser added a newline for no reason) and the
only fix was to put it at the beginning of the line.
 
>     Makia> By default, we found (via stats from the DDN) that we were
>     Makia> only seeing reads and writes in the 0-32Kbyte range.
>     Makia> Comparing IBGold and OFED, we found that the
>     Makia> srp_sg_tablesize defaulted to 256, but in OFED it defaulted
>     Makia> to 12.  So, changing this (via modprobe.conf) to 256 in
>     Makia> OFED, we were able to see reads and writes in the 128Kbyte
>     Makia> range (which is what ultimately got us to the performance
>     Makia> above).  I also noticed that there is a max_sects option
>     Makia> you can pass to add_target (in the SRP /sys entries) which
>     Makia> seemed to be the same idea as srp_sg_tablesize, but this
>     Makia> didn't seem to affect anything.
> 
> It is "max_sect" not "max_sects" (no final 's').  Anyway, what do you
> mean that it didn't affect anything?  max_sect=4096 should
> theoretically get you up to 512 KB IOs.

Sorry again about the type (I should never attempt to work off of memory).
With the max_sect=4096, and the srp_sg_tablesize to 256, we are now seeing
512KB IOs.  The new question is is there a way to get this to 1M IOs?
 
>  - R.

-- 
Makia Minich <minich at ornl.gov>
National Center for Computation Science
Oak Ridge National Laboratory