[openib-general] Srp question

[Roland Dreier]

    Makia> We are attempting to do some performance testing of the SRP driver (with a
    Makia> DDN target) and are seeing some poor results:

    Makia> ~120MB/s per lun with 1 sgp_dd
    Makia> ~80MB/s per lun with 4 sgp_dd

    Makia> Previously we had attempted the same tests with IBGold and got the
    Makia> following:

    Makia> ~150MB/s per lun with 1 sgp_dd
    Makia> ~600MB/s per lun with 4 sgp_dd

Were these tests with the same kernels otherwise?  If not, there may
be unrelated changes to the SCSI stack that affect synthetic
benchmarks like this.  (I seem to remember a change in the not too
distant patch that affected the largest IO it is possible to submit
through the SG interface).

    Makia> To achieve the results in IBGold, we were able to set the
    Makia> srp module option "max_xfer_sectors_per_io=4096", but can't
    Makia> seem to find an equivalent option in the OFED SRP drivers.

When connecting to the target (the echo to the add_target file), you
can add ",max_sect=4096" to the string you pass in.

    Makia> By default, we found (via stats from the DDN) that we were
    Makia> only seeing reads and writes in the 0-32Kbyte range.
    Makia> Comparing IBGold and OFED, we found that the
    Makia> srp_sg_tablesize defaulted to 256, but in OFED it defaulted
    Makia> to 12.  So, changing this (via modprobe.conf) to 256 in
    Makia> OFED, we were able to see reads and writes in the 128Kbyte
    Makia> range (which is what ultimately got us to the performance
    Makia> above).  I also noticed that there is a max_sects option
    Makia> you can pass to add_target (in the SRP /sys entries) which
    Makia> seemed to be the same idea as srp_sg_tablesize, but this
    Makia> didn't seem to affect anything.

It is "max_sect" not "max_sects" (no final 's').  Anyway, what do you
mean that it didn't affect anything?  max_sect=4096 should
theoretically get you up to 512 KB IOs.

 - R.