[ofa-general] WinOF_2_0_5/SRP initiator: slow reads and eventually hangs

[Chris Worley]

On Mon, Aug 10, 2009 at 4:40 AM, Bart Van
Assche<bart.vanassche at gmail.com> wrote:
> On Sun, Aug 9, 2009 at 7:09 PM, Chris Worley <worleys at gmail.com> wrote:
>>
>> I'm running a target comprised of: RHEL5.2/2.6.18-92.el5 (fresh off
>> the CD.. never updated) and it's embedded IB stack (not the latest
>> OFED) w/ SCST rev 1029 8-Aug-2009 ("svn info").
>>
>> I'm running a W2008S (fully patched) initiator w/
>> MLNX_WinOF_2_0_5_wlh_x64_fre_2_0_5_4453.
>>
>> Using Mellanox QDR cards/switch.
>>
>> Writes over SRP, as measured from the initiator using IOMeter, get
>> proper performance (i.e. 1.2GB/s).
>>
>> Reads get about 30% performance (i.e. 500MB/s instead of 1.6GB/s).
>> And while reading, IOMeter eventually hangs the system (Windows
>> becomes unresponsive to GUI interaction).  In this state, I see iostat
>> reporting transfers at the same low read rate from the target... so
>> there's IB traffic, but, given IOMeter's tasks are 10 minutes each, it
>> acts like it's a "skipping record" (sorry of you young folks don't
>> know what that is... but I can't think of another way to describe it)
>> and never moving on to the next benchmark, just endlessly repeating
>> the same I/O over and over again.  If I unload then reload the mlx4_ib
>> driver on the target, then the Windows system quickly returns, but
>> IOMeter remains hung and needs killed.
>
> The throughput of the SRP protocol strongly depends on the block size used
> for I/O. The results I obtained with IOmeter are:
> * For a block size of 32 KB: 396 MB/s for reading and 321 MB/s for writing.
> * For a block size of 1 MB: 1383 MB/s for reading and 1151 MB/s for writing.
> These results are about 90% of the throughput obtained with dd.
>
> Setup details:
> * Two Mellanox ConnectX DDR cards connected back to back, operating in PCIe
> 2.0 mode.
> * Target: vanilla 2.6.30.4 kernel + SCST patches + the two patches attached
> to http://bugzilla.kernel.org/show_bug.cgi?id=13757 + SCST r1030.
> * Initiator: openSUSE 11.0 (contains a patched 2.6.27.25 kernel) with
> openSUSE 11.0 OFED components + Linux version of IOmeter's dynamo + IOmeter
> GUI running in a virtual machine.
> * I/O-scheduler used by SRP initiator: noop.

Thanks for the recommendations (to both Bart and Joe).

I setup my target exactly as you prescribe... but my initiator is
still Windows (version of WInOF at top): performance as relayed by
IOMeter starts high and the average slowly decreases.  Watching the
instantaneous throughput, there seem to be longer and longer lags of
poor performance. between moments of good performance.  I need to run
this against a Linux initiator to see if the problems are w/ WinOF.

Using OFED 1.4.1 (w/ the stock RHEL kernel) on the target, the
performance was steady and getting close to acceptable.  In a 15 hour
test that cycles through sequential and random LBA's and R/W mixes
from block sizes from 1MB to 512B, it worked well and got decent
performance until it hit 1KB sequential reads which hung IOMeter; no
messages on the Linux side (all looked okay).  IBSRP on the Windows
side just said "a reset to device was issued" every 15 to 30 seconds
after the problem started. I reloaded the IB stack on the Linux side,
and was able to get it restarted.

Still a lot of combinations to test.

Thanks,

Chris
>
> Bart.
>