[ofa-general] low performance with multiple LUNs on a single portwith ib_srp
Canon, Richard Shane
canonrs at ornl.gov
Tue Jun 26 07:55:29 PDT 2007
Mahmoud,
Thanks for the hint. I tried that out and it definitely helped. The
key parameter is the max_cmd_per_lun. I think at 16 (which is what I
was using) it was overflowing something in the stack. I tried both 3
and 5. With 5 I was able to get over 700 MB/s for one up to four LUNs
on a single port. I was able to get 750 MB/s when using over two LUNs.
So that looks much better.
Thanks,
--Shane
________________________________
From: Mahmoud Hanafi [mailto:mhanafi at csc.com]
Sent: Tuesday, June 26, 2007 10:12 AM
To: Canon, Richard Shane
Cc: general at lists.openfabrics.org; Vu Pham
Subject: Re: [ofa-general] low performance with multiple LUNs on a
single portwith ib_srp
Here are some performance results that I was able to achieve running
across several Luns.
Config setting
1 host port to 1 ddn port.
OFED1.2rc6
[root at io1 IB]# cat /etc/modprobe.conf
alias scsi_hostadapter qla2xxx
alias scsi_hostadapter1 megaraid_sas
alias scsi_hostadapter2 qla2400
alias usb-controller ehci-hcd
alias usb-controller1 uhci-hcd
alias ib0 ib_ipoib
alias ib1 ib_ipoib
alias net-pf-27 ib_sdp
alias lustre llite
options lnet networks=o2ib
alias eth1 bnx2
alias eth0 bnx2
options ib_srp srp_sg_tablesize=256
[root at io1 IB]# cat /etc/srp_daemon.conf
a max_sect=8192,max_cmd_per_lun=3
Write (MB/sec)
Number of LUNS
"Rec Length
(KB)" 1 2 3 4 5 6
7
16 23 37 41 47 51
55 57
32 44 72 82 94 102
109 114
64 79 136 163 187
201 215 226
128 131 247 310 352
380 405 426
256 194 363 477 549
616 673 698
512 299 558 553 670
717 725 727
1,024 434 591 718 725
725 725 726
2,048 465 608 687 723
725 726 726
4,096 523 695 722 726
727 727 726
8,192 537 702 726 727
728 726 727
Read (MB/sec)
Number of LUNS
"Rec Length
(KB)" 1 2 3 4 5 6
7
16 26 41 45 56 60
63 62
32 48 78 97 107
117 122 124
64 81 140 172 196
215 227 237
128 126 207 269 314
347 373 391
256 174 271 389 482
500 537 546
512 255 375 418 478
528 556 562
1,024 330 430 505 554
564 564 564
2,048 326 445 527 553
561 563 564
4,096 357 513 556 562
564 564 565
8,192 360 520 558 564
565 565 565
------------------------------------------------------------------------
------------------------------------------------------------------------
--------------------------------
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit
written agreement or government initiative expressly permitting the use
of e-mail for such purpose.
------------------------------------------------------------------------
------------------------------------------------------------------------
--------------------------------
"Canon, Richard Shane" <canonrs at ornl.gov>
Sent by: general-bounces at lists.openfabrics.org
06/25/2007 05:51 PM
To
Vu Pham <vuhuong at mellanox.com>
cc
general at lists.openfabrics.org
Subject
[ofa-general] low performance with multiple LUNs on a single port
with ib_srp
Greetings,
Hopefully the subject says it all...
I've stumbled on a performance issue with the OFED ib_srp driver. Here
is the configuration. I am testing with a DDN 9550 and a single host
system. The systems are connected by two SDR links. On the host side
there is a dual port (DDR) card. On the DDN side, both lines go into a
single singlet (even though it is a couplet). The lines go into two
distinct cards on the DDN side (if you are familiar with the layout).
The testing used OFED 1.2.
Now for the tests... If I run a single stream test I'm seeing good
result with over 700 MB/s. These tests are run using sg_dd with the
directio flag. If I run two concurrent streams against two LUNs that
are each presented over a single port on the DDN (and therefore accessed
by a single port on the host side), the aggregate performance drop to
around 120 MB/s (60 MB/s per stream).
Just to confirm it isn't a problem on the DDN side, I repeated these
tests with the IBGD driver. There I consistently saw about 600-650 MB/s
on the port regardless of the number of LUNs I tested with.
Any ideas on what the problem is? Also, if this doesn't make sense, let
me know and I will try to clarify further.
Thanks,
--Shane Canon
--
R. Shane Canon
National Center for Computational Science
Oak Ridge National Laboratory
canonrs at ornl.gov
_______________________________________________
general mailing list
general at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20070626/e617e6b7/attachment.html>
More information about the general
mailing list