[ewg] iWARP uDAPL RDMA_READ(RR) test hangs on Chelsio T320

[Steve Wise]

Note: The iWARP wire protocol only supports 1 sink (and source) 
scatter-gather-entry.


On 6/11/2013 9:01 AM, John Jolly wrote:
> All,
>
> We're trying to track down a problem where dapltest hangs using more
> than one segment on the RR (RDMA read) performance test.
>
> Tried uDAPL RDMA_READ test with "number of segments" parameter set to 1, test
> worked fine.
>
> RDMA_WRITE and SEND/RECV tests were executed without any issue.
>
> ---Steps to Reproduce---
>   1. Configure iWARP on server and client machines connected via Chelsio
> adapter.
>
> Server configuration:
>
> x3755b:/ # ifconfig eth0
> eth0      Link encap:Ethernet  HWaddr 00:07:43:07:1B:B8
>            inet addr:10.0.0.1  Bcast:10.255.255.255  Mask:255.0.0.0
>            inet6 addr: fe80::207:43ff:fe07:1bb8/64 Scope:Link
>            UP BROADCAST MULTICAST  MTU:1500  Metric:1
>            RX packets:10011 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:9969 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:12629636 (12.0 Mb)  TX bytes:2510976 (2.3 Mb)
>            Interrupt:11 Memory:e7880000-e7880fff
>
> x3755b:/ # cat /etc/dat.conf
> ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth0 0" ""
>
> Client configuration:
>
> 3650m2:~ # ifconfig eth4
> eth4      Link encap:Ethernet  HWaddr 00:07:43:07:1B:62
>            inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.0.0.0
>            inet6 addr: fe80::207:43ff:fe07:1b62/64 Scope:Link
>            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>            RX packets:8730 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:8780 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:2407318 (2.2 Mb)  TX bytes:10847636 (10.3 Mb)
>            Interrupt:26 Memory:9c800000-9c800fff
>
>
> 3650m2:~ # cat /etc/dat.conf
> ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth4 0" ""
>
> 2. Run the below command on the server,
> x3755b:~ # dapltest -T S -D ofa-v2-iwarp
> Dapltest: Service Point Ready - ofa-v2-iwarp
> Dapltest: Service Point Ready - ofa-v2-iwarp
>
> 3. On client, run RR dapltest as below: Note the 2 parameter for number of segments.
> 3650m2:~ # dapltest -T P -s 10.0.0.1 -D ofa-v2-iwarp -d -i 100 RR 4096 2
> -------------------------------------
> PerfCmd.server_name              : 10.0.0.1
> PerfCmd.dapl_name                : ofa-v2-iwarp
> PerfCmd.mode                     : BLOCKING
> PerfCmd.num_iterations           : 100
> PerfCmd.pipeline_len             : -1
> PerfCmd.op.transfer_type         : RDMA_READ
> PerfCmd.op.num_segs              : 2
> PerfCmd.op.seg_size              : 4096
> Server Name: 10.0.0.1
> Server Net Address: 10.0.0.1
> DT_cs_Client: Starting Test ...
> DT_cs_Client: IA ofa-v2-iwarp opened
> DT_cs_Client: EP created
> *****  DAPL  Characteristics  *****
> Provider: ofa-v2-iwarp  Version 2.0  DAPL 2.0
> Adapter:  by  Version 2.0
> Supporting:
>      32736 EPs with 1023 DTOs and 8 RDMA/RDs each
>      32767 EVDs of up to 65536 entries  (default S/R size is 16/16)
>      IOVs of up to 4 elements
>      32768 LMRs (and 0 RMRs) of up to 0xffffffff bytes
>      Maximum MTU 0xffffffff bytes, RDMA 0xffffffff bytes
>      Maximum Private data size 256 bytes
>      Local IP address 10.0.0.2
> ***** ***** ***** ***** ***** *****
> DT_cs_Client: Posting 1 recv buffer
> DT_cs_Client: Connect Endpoint
> DT_cs_Client: Await connection ...
> DT_cs_Client: Connected!
> DAT_STATE: DAT_EP_STATE_CONNECTED
> DAT_STATE: Inbound DTO Status: Idle
> DAT_STATE: Outbound DTO Status: Active
> DT_cs_Client: Sending Client_Info
> DT_cs_Client: Sent Client_Info - awaiting completion
> DT_cs_Client: Sending Command
> DT_cs_Client: Sent Command - awaiting completion
> DT_cs_Client: Waiting for Server_Info
> DT_cs_Client: Server_Info Received
> DT_cs_Client: Version OK!
> -------------------------------------
> Server_Info.dapltest_version   : 6
> Server_Info.is_little_endian   : 1
> -------------------------------------
> Client_Info.dapltest_version   : 6
> Client_Info.is_little_endian   : 1
> Client_Info.test_type          : 2
> -------------------------------------
> PerfCmd.server_name              : 10.0.0.1
> PerfCmd.dapl_name                : ofa-v2-iwarp
> PerfCmd.mode                     : BLOCKING
> PerfCmd.num_iterations           : 100
> PerfCmd.pipeline_len             : -1
> PerfCmd.op.transfer_type         : RDMA_READ
> PerfCmd.op.num_segs              : 2
> PerfCmd.op.seg_size              : 4096
> Client: Starting performance test
> Client[b0e0]: Connect on port 0xb0e0
> Client[b0e0]: Got Connection
> Test[b0e0]: Waiting for Sync Msg
> Test[b0e0]: Sync Msg Received
> Test[b0e0]: Begin...
>
> <...test hangs...>
>
> Let me know if I can provide more information.
>
> Regards,
> John, SUSE
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg