[openib-general] Srp question

[Makia Minich]

We are attempting to do some performance testing of the SRP driver (with a
DDN target) and are seeing some poor results:

~120MB/s per lun with 1 sgp_dd
~80MB/s per lun with 4 sgp_dd

Previously we had attempted the same tests with IBGold and got the
following:

~150MB/s per lun with 1 sgp_dd
~600MB/s per lun with 4 sgp_dd

To achieve the results in IBGold, we were able to set the srp module option
"max_xfer_sectors_per_io=4096", but can't seem to find an equivalent option
in the OFED SRP drivers.

By default, we found (via stats from the DDN) that we were only seeing reads
and writes in the 0-32Kbyte range.  Comparing IBGold and OFED, we found that
the srp_sg_tablesize defaulted to 256, but in OFED it defaulted to 12.  So,
changing this (via modprobe.conf) to 256 in OFED, we were able to see reads
and writes in the 128Kbyte range (which is what ultimately got us to the
performance above).  I also noticed that there is a max_sects option you can
pass to add_target (in the SRP /sys entries) which seemed to be the same
idea as srp_sg_tablesize, but this didn't seem to affect anything.

So, my question is, what is the right magic to get SRP up to speed?

Thanks...

-- 
Makia Minich <minich at ornl.gov>
National Center for Computation Science
Oak Ridge National Laboratory