<br><font size=2 face="sans-serif">Here are some performance results that
I was able to achieve running across several Luns.</font>
<br>
<br><font size=2 face="sans-serif">Config setting</font>
<br><font size=2 face="sans-serif">1 host port to 1 ddn port.</font>
<br><font size=2 face="sans-serif">OFED1.2rc6</font>
<br>
<br><font size=2 face="sans-serif">[root@io1 IB]# cat /etc/modprobe.conf</font>
<br><font size=2 face="sans-serif">alias scsi_hostadapter qla2xxx</font>
<br><font size=2 face="sans-serif">alias scsi_hostadapter1 megaraid_sas</font>
<br><font size=2 face="sans-serif">alias scsi_hostadapter2 qla2400</font>
<br><font size=2 face="sans-serif">alias usb-controller ehci-hcd</font>
<br><font size=2 face="sans-serif">alias usb-controller1 uhci-hcd</font>
<br><font size=2 face="sans-serif">alias ib0 ib_ipoib</font>
<br><font size=2 face="sans-serif">alias ib1 ib_ipoib</font>
<br><font size=2 face="sans-serif">alias net-pf-27 ib_sdp</font>
<br><font size=2 face="sans-serif">alias lustre llite</font>
<br><font size=2 face="sans-serif">options lnet networks=o2ib</font>
<br><font size=2 face="sans-serif">alias eth1 bnx2</font>
<br><font size=2 face="sans-serif">alias eth0 bnx2</font>
<br><font size=2 face="sans-serif">options ib_srp srp_sg_tablesize=256<br>
<br>
[root@io1 IB]# cat /etc/srp_daemon.conf</font>
<br><font size=2 face="sans-serif">a max_sect=8192,max_cmd_per_lun=3</font>
<br>
<br>
<br>
<br><font size=2 face="sans-serif">
Write (MB/sec)
</font>
<br><font size=2 face="sans-serif">
Number of LUNS
</font>
<br><font size=2 face="sans-serif">"Rec Length</font>
<br><font size=2 face="sans-serif"> (KB)"
1
2 3
4 5
6 7</font>
<br><font size=2 face="sans-serif"> 16
23
37 41
47 51
55
57 </font>
<br><font size=2 face="sans-serif"> 32
44
72 82
94 102
109
114 </font>
<br><font size=2 face="sans-serif"> 64
79
136 163
187
201 215
226 </font>
<br><font size=2 face="sans-serif"> 128
131
247 310
352
380 405
426 </font>
<br><font size=2 face="sans-serif"> 256
194
363 477
549
616 673
698 </font>
<br><font size=2 face="sans-serif"> 512
299
558 553
670
717 725
727 </font>
<br><font size=2 face="sans-serif"> 1,024
434
591 718
725
725 725
726 </font>
<br><font size=2 face="sans-serif"> 2,048
465
608 687
723
725 726
726 </font>
<br><font size=2 face="sans-serif"> 4,096
523
695 722
726
727 727
726 </font>
<br><font size=2 face="sans-serif"> 8,192
537
702 726
727
728 726
727 </font>
<br>
<br><font size=2 face="sans-serif">
Read (MB/sec)
</font>
<br><font size=2 face="sans-serif">
Number
of LUNS
</font>
<br><font size=2 face="sans-serif">"Rec Length</font>
<br><font size=2 face="sans-serif"> (KB)"
1
2 3
4 5
6 7</font>
<br><font size=2 face="sans-serif"> 16
26
41 45
56 60
63
62 </font>
<br><font size=2 face="sans-serif"> 32
48
78 97
107 117
122
124 </font>
<br><font size=2 face="sans-serif"> 64
81
140 172
196
215 227
237 </font>
<br><font size=2 face="sans-serif"> 128
126
207 269
314
347 373
391 </font>
<br><font size=2 face="sans-serif"> 256
174
271 389
482
500 537
546 </font>
<br><font size=2 face="sans-serif"> 512
255
375 418
478
528 556
562 </font>
<br><font size=2 face="sans-serif"> 1,024
330
430 505
554
564 564
564 </font>
<br><font size=2 face="sans-serif"> 2,048
326
445 527
553
561 563
564 </font>
<br><font size=2 face="sans-serif"> 4,096
357
513 556
562
564 564
565 </font>
<br><font size=2 face="sans-serif"> 8,192
360
520 558
564
565 565
565 <br>
<br>
<br>
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit written
agreement or government initiative expressly permitting the use of e-mail
for such purpose.<br>
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>"Canon, Richard Shane"
<canonrs@ornl.gov></b> </font>
<br><font size=1 face="sans-serif">Sent by: general-bounces@lists.openfabrics.org</font>
<p><font size=1 face="sans-serif">06/25/2007 05:51 PM</font>
<td width=59%>
<table width=100%>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td><font size=1 face="sans-serif">Vu Pham <vuhuong@mellanox.com></font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td><font size=1 face="sans-serif">general@lists.openfabrics.org</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">[ofa-general] low performance with multiple
LUNs on a single port with ib_srp</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Greetings,</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Hopefully the subject says it all…</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">I’ve stumbled on a performance issue with
the OFED ib_srp driver. Here is the configuration. I am testing
with a DDN 9550 and a single host system. The systems are connected
by two SDR links. On the host side there is a dual port (DDR) card.
On the DDN side, both lines go into a single singlet (even though
it is a couplet). The lines go into two distinct cards on the DDN
side (if you are familiar with the layout). The testing used OFED
1.2.</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Now for the tests… If I run a single
stream test I’m seeing good result with over 700 MB/s. These tests
are run using sg_dd with the directio flag. If I run two concurrent
streams against two LUNs that are each presented over a single port on
the DDN (and therefore accessed by a single port on the host side), the
aggregate performance drop to around 120 MB/s (60 MB/s per stream).</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Just to confirm it isn’t a problem on the
DDN side, I repeated these tests with the IBGD driver. There I consistently
saw about 600-650 MB/s on the port regardless of the number of LUNs I tested
with.</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Any ideas on what the problem is? Also,
if this doesn’t make sense, let me know and I will try to clarify further.</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Thanks,</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">--Shane Canon</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">--</font>
<br><font size=2 face="Arial">R. Shane Canon</font>
<br><font size=2 face="Arial">National Center for Computational Science</font>
<br><font size=2 face="Arial">Oak Ridge National Laboratory</font>
<br><font size=2 face="Arial">canonrs@ornl.gov</font>
<br><font size=3 face="Times New Roman"> </font><font size=2><tt>_______________________________________________<br>
general mailing list<br>
general@lists.openfabrics.org<br>
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general<br>
<br>
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general</tt></font>
<br>