[ewg] iWARP uDAPL RDMA_READ(RR) test hangs on Chelsio T320

[John Jolly]

All,

We're trying to track down a problem where dapltest hangs using more
than one segment on the RR (RDMA read) performance test.

Tried uDAPL RDMA_READ test with "number of segments" parameter set to 1, test
worked fine.

RDMA_WRITE and SEND/RECV tests were executed without any issue.

---Steps to Reproduce---
 1. Configure iWARP on server and client machines connected via Chelsio
adapter.

Server configuration:

x3755b:/ # ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:07:43:07:1B:B8  
          inet addr:10.0.0.1  Bcast:10.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::207:43ff:fe07:1bb8/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:10011 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9969 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:12629636 (12.0 Mb)  TX bytes:2510976 (2.3 Mb)
          Interrupt:11 Memory:e7880000-e7880fff 

x3755b:/ # cat /etc/dat.conf 
ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth0 0" ""

Client configuration:

3650m2:~ # ifconfig eth4
eth4      Link encap:Ethernet  HWaddr 00:07:43:07:1B:62  
          inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::207:43ff:fe07:1b62/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8730 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8780 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2407318 (2.2 Mb)  TX bytes:10847636 (10.3 Mb)
          Interrupt:26 Memory:9c800000-9c800fff


3650m2:~ # cat /etc/dat.conf 
ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth4 0" ""

2. Run the below command on the server,
x3755b:~ # dapltest -T S -D ofa-v2-iwarp
Dapltest: Service Point Ready - ofa-v2-iwarp
Dapltest: Service Point Ready - ofa-v2-iwarp

3. On client, run RR dapltest as below: Note the 2 parameter for number of segments.
3650m2:~ # dapltest -T P -s 10.0.0.1 -D ofa-v2-iwarp -d -i 100 RR 4096 2
-------------------------------------
PerfCmd.server_name              : 10.0.0.1
PerfCmd.dapl_name                : ofa-v2-iwarp
PerfCmd.mode                     : BLOCKING
PerfCmd.num_iterations           : 100
PerfCmd.pipeline_len             : -1
PerfCmd.op.transfer_type         : RDMA_READ
PerfCmd.op.num_segs              : 2
PerfCmd.op.seg_size              : 4096
Server Name: 10.0.0.1 
Server Net Address: 10.0.0.1
DT_cs_Client: Starting Test ... 
DT_cs_Client: IA ofa-v2-iwarp opened
DT_cs_Client: EP created
*****  DAPL  Characteristics  *****
Provider: ofa-v2-iwarp  Version 2.0  DAPL 2.0
Adapter:  by  Version 2.0
Supporting:
    32736 EPs with 1023 DTOs and 8 RDMA/RDs each
    32767 EVDs of up to 65536 entries  (default S/R size is 16/16)
    IOVs of up to 4 elements
    32768 LMRs (and 0 RMRs) of up to 0xffffffff bytes
    Maximum MTU 0xffffffff bytes, RDMA 0xffffffff bytes
    Maximum Private data size 256 bytes
    Local IP address 10.0.0.2
***** ***** ***** ***** ***** *****
DT_cs_Client: Posting 1 recv buffer
DT_cs_Client: Connect Endpoint
DT_cs_Client: Await connection ...
DT_cs_Client: Connected!
DAT_STATE: DAT_EP_STATE_CONNECTED
DAT_STATE: Inbound DTO Status: Idle 
DAT_STATE: Outbound DTO Status: Active
DT_cs_Client: Sending Client_Info
DT_cs_Client: Sent Client_Info - awaiting completion
DT_cs_Client: Sending Command
DT_cs_Client: Sent Command - awaiting completion
DT_cs_Client: Waiting for Server_Info
DT_cs_Client: Server_Info Received
DT_cs_Client: Version OK!
-------------------------------------
Server_Info.dapltest_version   : 6
Server_Info.is_little_endian   : 1
-------------------------------------
Client_Info.dapltest_version   : 6
Client_Info.is_little_endian   : 1
Client_Info.test_type          : 2
-------------------------------------
PerfCmd.server_name              : 10.0.0.1
PerfCmd.dapl_name                : ofa-v2-iwarp
PerfCmd.mode                     : BLOCKING
PerfCmd.num_iterations           : 100
PerfCmd.pipeline_len             : -1
PerfCmd.op.transfer_type         : RDMA_READ
PerfCmd.op.num_segs              : 2
PerfCmd.op.seg_size              : 4096
Client: Starting performance test
Client[b0e0]: Connect on port 0xb0e0
Client[b0e0]: Got Connection
Test[b0e0]: Waiting for Sync Msg
Test[b0e0]: Sync Msg Received
Test[b0e0]: Begin...

<...test hangs...>

Let me know if I can provide more information.

Regards,
John, SUSE