[ewg] iWARP uDAPL RDMA_READ(RR) test hangs on Chelsio T320
Steve Wise
swise at opengridcomputing.com
Tue Jun 11 12:26:01 PDT 2013
Is this T3 or T4?
If dapl calls ibv_post_send() with wr->num_sge > 1 for a read request,
the chelsio provider will return -EINVAL in T4 and -1 in T3.
We shouldn't see a hang though. :)
On 6/11/2013 2:18 PM, Steve Wise wrote:
> Note: The iWARP wire protocol only supports 1 sink (and source)
> scatter-gather-entry.
>
>
> On 6/11/2013 9:01 AM, John Jolly wrote:
>> All,
>>
>> We're trying to track down a problem where dapltest hangs using more
>> than one segment on the RR (RDMA read) performance test.
>>
>> Tried uDAPL RDMA_READ test with "number of segments" parameter set to
>> 1, test
>> worked fine.
>>
>> RDMA_WRITE and SEND/RECV tests were executed without any issue.
>>
>> ---Steps to Reproduce---
>> 1. Configure iWARP on server and client machines connected via Chelsio
>> adapter.
>>
>> Server configuration:
>>
>> x3755b:/ # ifconfig eth0
>> eth0 Link encap:Ethernet HWaddr 00:07:43:07:1B:B8
>> inet addr:10.0.0.1 Bcast:10.255.255.255 Mask:255.0.0.0
>> inet6 addr: fe80::207:43ff:fe07:1bb8/64 Scope:Link
>> UP BROADCAST MULTICAST MTU:1500 Metric:1
>> RX packets:10011 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:9969 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:12629636 (12.0 Mb) TX bytes:2510976 (2.3 Mb)
>> Interrupt:11 Memory:e7880000-e7880fff
>>
>> x3755b:/ # cat /etc/dat.conf
>> ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0
>> "eth0 0" ""
>>
>> Client configuration:
>>
>> 3650m2:~ # ifconfig eth4
>> eth4 Link encap:Ethernet HWaddr 00:07:43:07:1B:62
>> inet addr:10.0.0.2 Bcast:10.255.255.255 Mask:255.0.0.0
>> inet6 addr: fe80::207:43ff:fe07:1b62/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:8730 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:8780 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:2407318 (2.2 Mb) TX bytes:10847636 (10.3 Mb)
>> Interrupt:26 Memory:9c800000-9c800fff
>>
>>
>> 3650m2:~ # cat /etc/dat.conf
>> ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0
>> "eth4 0" ""
>>
>> 2. Run the below command on the server,
>> x3755b:~ # dapltest -T S -D ofa-v2-iwarp
>> Dapltest: Service Point Ready - ofa-v2-iwarp
>> Dapltest: Service Point Ready - ofa-v2-iwarp
>>
>> 3. On client, run RR dapltest as below: Note the 2 parameter for
>> number of segments.
>> 3650m2:~ # dapltest -T P -s 10.0.0.1 -D ofa-v2-iwarp -d -i 100 RR 4096 2
>> -------------------------------------
>> PerfCmd.server_name : 10.0.0.1
>> PerfCmd.dapl_name : ofa-v2-iwarp
>> PerfCmd.mode : BLOCKING
>> PerfCmd.num_iterations : 100
>> PerfCmd.pipeline_len : -1
>> PerfCmd.op.transfer_type : RDMA_READ
>> PerfCmd.op.num_segs : 2
>> PerfCmd.op.seg_size : 4096
>> Server Name: 10.0.0.1
>> Server Net Address: 10.0.0.1
>> DT_cs_Client: Starting Test ...
>> DT_cs_Client: IA ofa-v2-iwarp opened
>> DT_cs_Client: EP created
>> ***** DAPL Characteristics *****
>> Provider: ofa-v2-iwarp Version 2.0 DAPL 2.0
>> Adapter: by Version 2.0
>> Supporting:
>> 32736 EPs with 1023 DTOs and 8 RDMA/RDs each
>> 32767 EVDs of up to 65536 entries (default S/R size is 16/16)
>> IOVs of up to 4 elements
>> 32768 LMRs (and 0 RMRs) of up to 0xffffffff bytes
>> Maximum MTU 0xffffffff bytes, RDMA 0xffffffff bytes
>> Maximum Private data size 256 bytes
>> Local IP address 10.0.0.2
>> ***** ***** ***** ***** ***** *****
>> DT_cs_Client: Posting 1 recv buffer
>> DT_cs_Client: Connect Endpoint
>> DT_cs_Client: Await connection ...
>> DT_cs_Client: Connected!
>> DAT_STATE: DAT_EP_STATE_CONNECTED
>> DAT_STATE: Inbound DTO Status: Idle
>> DAT_STATE: Outbound DTO Status: Active
>> DT_cs_Client: Sending Client_Info
>> DT_cs_Client: Sent Client_Info - awaiting completion
>> DT_cs_Client: Sending Command
>> DT_cs_Client: Sent Command - awaiting completion
>> DT_cs_Client: Waiting for Server_Info
>> DT_cs_Client: Server_Info Received
>> DT_cs_Client: Version OK!
>> -------------------------------------
>> Server_Info.dapltest_version : 6
>> Server_Info.is_little_endian : 1
>> -------------------------------------
>> Client_Info.dapltest_version : 6
>> Client_Info.is_little_endian : 1
>> Client_Info.test_type : 2
>> -------------------------------------
>> PerfCmd.server_name : 10.0.0.1
>> PerfCmd.dapl_name : ofa-v2-iwarp
>> PerfCmd.mode : BLOCKING
>> PerfCmd.num_iterations : 100
>> PerfCmd.pipeline_len : -1
>> PerfCmd.op.transfer_type : RDMA_READ
>> PerfCmd.op.num_segs : 2
>> PerfCmd.op.seg_size : 4096
>> Client: Starting performance test
>> Client[b0e0]: Connect on port 0xb0e0
>> Client[b0e0]: Got Connection
>> Test[b0e0]: Waiting for Sync Msg
>> Test[b0e0]: Sync Msg Received
>> Test[b0e0]: Begin...
>>
>> <...test hangs...>
>>
>> Let me know if I can provide more information.
>>
>> Regards,
>> John, SUSE
>> _______________________________________________
>> ewg mailing list
>> ewg at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
More information about the ewg
mailing list