[ofa-general] low performance with multiple LUNs on a single portwith ib_srp

Canon, Richard Shane canonrs at ornl.gov
Tue Jun 26 07:55:29 PDT 2007


 

Mahmoud,

 

Thanks for the hint.  I tried that out and it definitely helped.  The
key parameter is the max_cmd_per_lun.  I think at 16 (which is what I
was using) it was overflowing something in the stack.  I tried both 3
and 5.  With 5 I was able to get over 700 MB/s for one up to four LUNs
on a single port.  I was able to get 750 MB/s when using over two LUNs.
So that looks much better.

 

Thanks,

 

--Shane

 

________________________________

From: Mahmoud Hanafi [mailto:mhanafi at csc.com] 
Sent: Tuesday, June 26, 2007 10:12 AM
To: Canon, Richard Shane
Cc: general at lists.openfabrics.org; Vu Pham
Subject: Re: [ofa-general] low performance with multiple LUNs on a
single portwith ib_srp

 


Here are some performance results that I was able to achieve running
across several Luns. 

Config setting 
1 host port to 1 ddn port. 
OFED1.2rc6 

[root at io1 IB]# cat /etc/modprobe.conf 
alias scsi_hostadapter qla2xxx 
alias scsi_hostadapter1 megaraid_sas 
alias scsi_hostadapter2 qla2400 
alias usb-controller ehci-hcd 
alias usb-controller1 uhci-hcd 
alias ib0 ib_ipoib 
alias ib1 ib_ipoib 
alias net-pf-27 ib_sdp 
alias lustre llite 
options lnet networks=o2ib 
alias eth1 bnx2 
alias eth0 bnx2 
options ib_srp srp_sg_tablesize=256

[root at io1 IB]# cat /etc/srp_daemon.conf 
a max_sect=8192,max_cmd_per_lun=3 



                Write (MB/sec)

                Number of LUNS

"Rec Length 
 (KB)"                1        2        3        4        5        6
7 
 16                  23          37          41          47          51
55          57 
 32                  44          72          82          94          102
109          114 
 64                  79          136          163          187
201          215          226 
 128                  131          247          310          352
380          405          426 
 256                  194          363          477          549
616          673          698 
 512                  299          558          553          670
717          725          727 
 1,024                  434          591          718          725
725          725          726 
 2,048                  465          608          687          723
725          726          726 
 4,096                  523          695          722          726
727          727          726 
 8,192                  537          702          726          727
728          726          727 

                        Read (MB/sec)

                        Number of LUNS

"Rec Length 
 (KB)"                1        2        3        4        5        6
7 
 16                  26          41          45          56          60
63          62 
 32                  48          78          97          107
117          122          124 
 64                  81          140          172          196
215          227          237 
 128                  126          207          269          314
347          373          391 
 256                  174          271          389          482
500          537          546 
 512                  255          375          418          478
528          556          562 
 1,024                  330          430          505          554
564          564          564 
 2,048                  326          445          527          553
561          563          564 
 4,096                  357          513          556          562
564          564          565 
 8,192                  360          520          558          564
565          565          565 


------------------------------------------------------------------------
------------------------------------------------------------------------
--------------------------------
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit
written agreement or government initiative expressly permitting the use
of e-mail for such purpose.
------------------------------------------------------------------------
------------------------------------------------------------------------
--------------------------------




"Canon, Richard Shane" <canonrs at ornl.gov> 
Sent by: general-bounces at lists.openfabrics.org 

06/25/2007 05:51 PM 

To

Vu Pham <vuhuong at mellanox.com> 

cc

general at lists.openfabrics.org 

Subject

[ofa-general] low performance with multiple LUNs on a single port
with ib_srp

 

 

 




  
Greetings, 
  
Hopefully the subject says it all... 
  
I've stumbled on a performance issue with the OFED ib_srp driver.  Here
is the configuration.  I am testing with a DDN 9550 and a single host
system.  The systems are connected by two SDR links.  On the host side
there is a dual port (DDR) card.  On the DDN side, both lines go into a
single singlet (even though it is a couplet).  The lines go into two
distinct cards on the DDN side (if you are familiar with the layout).
The testing used OFED 1.2. 
  
Now for the tests...  If I run a single stream test I'm seeing good
result with over 700 MB/s.  These tests are run using sg_dd with the
directio flag.  If I run two concurrent streams against two LUNs that
are each presented over a single port on the DDN (and therefore accessed
by a single port on the host side), the aggregate performance drop to
around 120 MB/s (60 MB/s per stream). 
  
Just to confirm it isn't a problem on the DDN side, I repeated these
tests with the IBGD driver.  There I consistently saw about 600-650 MB/s
on the port regardless of the number of LUNs I tested with. 
  
Any ideas on what the problem is?  Also, if this doesn't make sense, let
me know and I will try to clarify further. 
  
Thanks, 
  
--Shane Canon 
  
  
  
-- 
R. Shane Canon 
National Center for Computational Science 
Oak Ridge National Laboratory 
canonrs at ornl.gov 
 _______________________________________________
general mailing list
general at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20070626/e617e6b7/attachment.html>


More information about the general mailing list