[ofa-general] WinOF_2_0_5/SRP initiator: slow reads and eventually hangs

[Bart Van Assche]

On Sun, Aug 9, 2009 at 7:09 PM, Chris Worley <worleys at gmail.com> wrote:

> I'm running a target comprised of: RHEL5.2/2.6.18-92.el5 (fresh off
> the CD.. never updated) and it's embedded IB stack (not the latest
> OFED) w/ SCST rev 1029 8-Aug-2009 ("svn info").
>
> I'm running a W2008S (fully patched) initiator w/
> MLNX_WinOF_2_0_5_wlh_x64_fre_2_0_5_4453.
>
> Using Mellanox QDR cards/switch.
>
> Writes over SRP, as measured from the initiator using IOMeter, get
> proper performance (i.e. 1.2GB/s).
>
> Reads get about 30% performance (i.e. 500MB/s instead of 1.6GB/s).
> And while reading, IOMeter eventually hangs the system (Windows
> becomes unresponsive to GUI interaction).  In this state, I see iostat
> reporting transfers at the same low read rate from the target... so
> there's IB traffic, but, given IOMeter's tasks are 10 minutes each, it
> acts like it's a "skipping record" (sorry of you young folks don't
> know what that is... but I can't think of another way to describe it)
> and never moving on to the next benchmark, just endlessly repeating
> the same I/O over and over again.  If I unload then reload the mlx4_ib
> driver on the target, then the Windows system quickly returns, but
> IOMeter remains hung and needs killed.
>

The throughput of the SRP protocol strongly depends on the block size used
for I/O. The results I obtained with IOmeter are:
* For a block size of 32 KB: 396 MB/s for reading and 321 MB/s for writing.
* For a block size of 1 MB: 1383 MB/s for reading and 1151 MB/s for writing.
These results are about 90% of the throughput obtained with dd.

Setup details:
* Two Mellanox ConnectX DDR cards connected back to back, operating in PCIe
2.0 mode.
* Target: vanilla 2.6.30.4 kernel + SCST patches + the two patches attached
to http://bugzilla.kernel.org/show_bug.cgi?id=13757 + SCST r1030.
* Initiator: openSUSE 11.0 (contains a patched 2.6.27.25 kernel) with
openSUSE 11.0 OFED components + Linux version of IOmeter's dynamo + IOmeter
GUI running in a virtual machine.
* I/O-scheduler used by SRP initiator: noop.

Bart.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090810/d70a5210/attachment.html>