***SPAM*** Re: [ofa-general] Re: iSer tuning guide?

[Chris Worley]

On Fri, Apr 10, 2009 at 3:34 PM, Chris Worley <worleys at gmail.com> wrote:
> It looks like it's the target side at fault.  The target in stalled is
> the one pointed to by the OFED wiki:
>
> http://www.voltaire.com/ftp/support-products/source/stgt/scsi-target-utils-0.1-20080828.x86_64.rpm
>
> ... it's mind-boggling to believe that any OFED component that's 8
> months old would be compatible with the latest release... but that
> directory isn't browsable, so it's hard to tell if there is something
> newer.

The OFED iSer wiki does say "1.4 includes iSer target support"... so
rather than using the rpm shown above, or the install ofed.conf also
referred to by the twiki, I configured OFED w/ tgt (somebody w/
permission should fix the wiki).  It created two conflicting RPMs:

scsi-target-utils-0.1-20080828.x86_64.rpm

and

tgt-0.1-20080828.x86_64.rpm

... both had the same issues /w iSer as previously reported (one target max).

Note that for discovery I use:

iscsi_discovery <IP> -t iser -f -l

If I use "tcp" instead of "iser", then multiple targets work, but
<400MB/s... over Quad IB w/ drives that get >GB/s.

In looking around the web and at other mailing lists, it looks like
iSer is still in it's infancy and there is no reliable IB
implementation, which would be be exemplified by this one-sided
conversation.

Chris
>
> Most of the time, if I restart the tgtd and opensm/openibd on the
> target system, then restart just iscsi on the initiators, I can get
> iser working again after it hangs... but sometimes the tgtd gets into
> a state where it segfaults at launch, and I have to reboot the target
> machine before things work again.
>
> It looks like it works great with one initiator and one target... any
> more than that and the connections are hung and the target needs
> restarted.
>
> I can run "fio" with one thread against two targets, and it's okay...
> more than one thread, and it hangs.
>
> Multiple initiators (separate systems) work as long as the access is
> serial... accessing two different or the same target from two
> different initiators simultaneously, and it hangs.
>
> I have seen it crash the initiator also, but no console to see what
> the crash was.
>
> So, I guess I'll just start trying recent OFED versions until I find a
> stable iser. If anybody can give me a hint as to what might be both
> stable and perform well, it would be appreciated.
>
> Thanks,
>
> Chris
> On Fri, Apr 10, 2009 at 2:35 PM, Chris Worley <worleys at gmail.com> wrote:
>> What's version of OFED has both a stable and high performance iSer?
>>
>> I setup the latest OFED (1.4.1rc3), and the speed kicked in, but using
>> fio on the initiator using two targets hung quickly on the initiator:
>>
>>  connection3:0: ping timeout of 5 secs expired, last rx 4557795905,
>> last ping 4557800905, now 4557805905
>>  connection3:0: detected conn error (1011)
>> iser: iscsi_iser_ep_disconnect:ib conn ffff8104dc482ec0 state 2
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>>  connection4:0: ping timeout of 5 secs expired, last rx 4557796701,
>> last ping 4557801701, now 4557806701
>>  connection4:0: detected conn error (1011)
>>  connection1:0: ping timeout of 5 secs expired, last rx 4557799350,
>> last ping 4557804350, now 4557809350
>>  connection1:0: detected conn error (1011)
>>  connection2:0: ping timeout of 5 secs expired, last rx 4557800108,
>> last ping 4557805108, now 4557810108
>>  connection2:0: detected conn error (1011)
>>  connection6:0: ping timeout of 5 secs expired, last rx 4557800224,
>> last ping 4557805224, now 4557810224
>>  connection6:0: detected conn error (1011)
>>  connection5:0: ping timeout of 5 secs expired, last rx 4557800231,
>> last ping 4557805231, now 4557810234
>>  connection5:0: detected conn error (1011)
>> iser: iser_cma_handler:event 10 conn ffff8104dc482ec0 id ffff8105b1502800
>> iser: iser_free_ib_conn_res:freeing conn ffff8104dc482ec0 cma_id
>> ffff8105b1502800 fmr pool ffff8105f9b555c0 qp ffff81027b346000
>> iser: iser_device_try_release:device ffff81081bb81640 refcount 5
>> iser: iscsi_iser_ep_disconnect:ib conn ffff8104dc4824c0 state 2
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>>  session3: session recovery timed out after 120 secs
>> iser: iser_cma_handler:event 10 conn ffff8104dc4824c0 id ffff8107df2ce400
>> iser: iser_free_ib_conn_res:freeing conn ffff8104dc4824c0 cma_id
>> ffff8107df2ce400 fmr pool ffff8107f60c90c0 qp ffff8107720d2400
>> iser: iser_device_try_release:device ffff81081bb81640 refcount 4
>> iser: iscsi_iser_ep_disconnect:ib conn ffff81081fa521c0 state 2
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>> iser: iser_cq_tasklet_fn:comp w. error op 0 status 5
>>  session4: session recovery timed out after 120 secs
>>
>>
>> On Fri, Apr 10, 2009 at 8:53 AM, Chris Worley <worleys at gmail.com> wrote:
>>> I'm running RHEL5.2 w/ the stock OFED RPMs.  I have a target disk that
>>> run at 800MB/s locally and QDR IB.  I need to get some performance out
>>> of this, as I'm seeing <300MB/s in my benchmarks.
>>>
>>> On the initiator side, I run a test using 1MB block sizes.
>>>
>>> On the target side I'm seeing everything in 4K packets.
>>>
>>> Any idea who's setting this and why?
>>>
>>> Is there a tuning guide available?
>>>
>>> Thanks,
>>>
>>> Chris
>>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>>
>