[ofa-general] low performance with multiple LUNs on a single port with ib_srp

Tue Jun 26 07:12:27 PDT 2007

Here are some performance results that I was able to achieve running 
across several Luns.

Config setting
1 host port to 1 ddn port.
OFED1.2rc6

[root at io1 IB]# cat /etc/modprobe.conf
alias scsi_hostadapter qla2xxx
alias scsi_hostadapter1 megaraid_sas
alias scsi_hostadapter2 qla2400
alias usb-controller ehci-hcd
alias usb-controller1 uhci-hcd
alias ib0 ib_ipoib
alias ib1 ib_ipoib
alias net-pf-27 ib_sdp
alias lustre llite
options lnet networks=o2ib
alias eth1 bnx2
alias eth0 bnx2
options ib_srp srp_sg_tablesize=256

[root at io1 IB]# cat /etc/srp_daemon.conf
a max_sect=8192,max_cmd_per_lun=3

                Write (MB/sec) 
                Number of LUNS 
"Rec Length
 (KB)"          1       2       3       4       5       6       7
 16              23      37      41      47      51      55      57 
 32              44      72      82      94      102     109     114 
 64              79      136     163     187     201     215     226 
 128             131     247     310     352     380     405     426 
 256             194     363     477     549     616     673     698 
 512             299     558     553     670     717     725     727 
 1,024           434     591     718     725     725     725     726 
 2,048           465     608     687     723     725     726     726 
 4,096           523     695     722     726     727     727     726 
 8,192           537     702     726     727     728     726     727 

                        Read (MB/sec)  
                        Number of LUNS 
"Rec Length
 (KB)"          1       2       3       4       5       6       7
 16              26      41      45      56      60      63      62 
 32              48      78      97      107     117     122     124 
 64              81      140     172     196     215     227     237 
 128             126     207     269     314     347     373     391 
 256             174     271     389     482     500     537     546 
 512             255     375     418     478     528     556     562 
 1,024           330     430     505     554     564     564     564 
 2,048           326     445     527     553     561     563     564 
 4,096           357     513     556     562     564     564     565 
 8,192           360     520     558     564     565     565     565 

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This is a PRIVATE message. If you are not the intended recipient, please 
delete without copying and kindly advise us by e-mail of the mistake in 
delivery. NOTE: Regardless of content, this e-mail shall not operate to 
bind CSC to any order or other contract unless pursuant to explicit 
written agreement or government initiative expressly permitting the use of 
e-mail for such purpose.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

"Canon, Richard Shane" <canonrs at ornl.gov> 
Sent by: general-bounces at lists.openfabrics.org
06/25/2007 05:51 PM

To
Vu Pham <vuhuong at mellanox.com>
cc
general at lists.openfabrics.org
Subject
[ofa-general] low performance with multiple LUNs on a single port with 
ib_srp

Greetings,

Hopefully the subject says it all…

I’ve stumbled on a performance issue with the OFED ib_srp driver.  Here is 
the configuration.  I am testing with a DDN 9550 and a single host system. 
 The systems are connected by two SDR links.  On the host side there is a 
dual port (DDR) card.  On the DDN side, both lines go into a single 
singlet (even though it is a couplet).  The lines go into two distinct 
cards on the DDN side (if you are familiar with the layout).  The testing 
used OFED 1.2.

Now for the tests…  If I run a single stream test I’m seeing good result 
with over 700 MB/s.  These tests are run using sg_dd with the directio 
flag.  If I run two concurrent streams against two LUNs that are each 
presented over a single port on the DDN (and therefore accessed by a 
single port on the host side), the aggregate performance drop to around 
120 MB/s (60 MB/s per stream).

Just to confirm it isn’t a problem on the DDN side, I repeated these tests 
with the IBGD driver.  There I consistently saw about 600-650 MB/s on the 
port regardless of the number of LUNs I tested with.

Any ideas on what the problem is?  Also, if this doesn’t make sense, let 
me know and I will try to clarify further.

Thanks,

--Shane Canon

--
R. Shane Canon
National Center for Computational Science
Oak Ridge National Laboratory
canonrs at ornl.gov
 _______________________________________________
general mailing list
general at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit 
http://openib.org/mailman/listinfo/openib-general

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20070626/335134f4/attachment.html>