[ewg] iWARP uDAPL RDMA_READ(RR) test hangs on Chelsio T320
John Jolly
jjolly at suse.com
Wed Jun 12 15:45:12 PDT 2013
Steve,
On Tue, Jun 11, 2013 at 02:26:01PM -0500, Steve Wise wrote:
> Is this T3 or T4?
22:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter
Subsystem: Chelsio Communications Inc Device 0001
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at e7880000 (64-bit, non-prefetchable) [size=4K]
Memory at e7000000 (64-bit, non-prefetchable) [size=8M]
Memory at e7881000 (64-bit, non-prefetchable) [size=4K]
[virtual] Expansion ROM at e6000000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [48] Message Signalled Interrupts: Mask- 64bit+ Count=1/32 Enable-
Capabilities: [58] Express Endpoint, MSI 00
Capabilities: [94] Vital Product Data <?>
Capabilities: [9c] MSI-X: Enable+ Mask- TabSize=32
Capabilities: [100] Device Serial Number 01-00-00-00-01-00-00-00
Capabilities: [300] Advanced Error Reporting
Kernel driver in use: cxgb3
Kernel modules: cxgb3
>
> If dapl calls ibv_post_send() with wr->num_sge > 1 for a read
> request, the chelsio provider will return -EINVAL in T4 and -1 in
> T3.
So is this a matter of telling our customer to stop doing this because
it's not supported?
>
> We shouldn't see a hang though. :)
We are getting reports of other dapltest problems, such as
'!(t4_rq_empty(wq))' assertion on a pair of T422s. I have the feeling
I will become very familiar with dapltest.
>
> On 6/11/2013 2:18 PM, Steve Wise wrote:
> >Note: The iWARP wire protocol only supports 1 sink (and source)
> >scatter-gather-entry.
> >
> >
> >On 6/11/2013 9:01 AM, John Jolly wrote:
> >>All,
> >>
> >>We're trying to track down a problem where dapltest hangs using more
> >>than one segment on the RR (RDMA read) performance test.
> >>
> >>Tried uDAPL RDMA_READ test with "number of segments" parameter
> >>set to 1, test
> >>worked fine.
> >>
> >>RDMA_WRITE and SEND/RECV tests were executed without any issue.
> >>
> >>---Steps to Reproduce---
> >> 1. Configure iWARP on server and client machines connected via Chelsio
> >>adapter.
> >>
> >>Server configuration:
> >>
> >>x3755b:/ # ifconfig eth0
> >>eth0 Link encap:Ethernet HWaddr 00:07:43:07:1B:B8
> >> inet addr:10.0.0.1 Bcast:10.255.255.255 Mask:255.0.0.0
> >> inet6 addr: fe80::207:43ff:fe07:1bb8/64 Scope:Link
> >> UP BROADCAST MULTICAST MTU:1500 Metric:1
> >> RX packets:10011 errors:0 dropped:0 overruns:0 frame:0
> >> TX packets:9969 errors:0 dropped:0 overruns:0 carrier:0
> >> collisions:0 txqueuelen:1000
> >> RX bytes:12629636 (12.0 Mb) TX bytes:2510976 (2.3 Mb)
> >> Interrupt:11 Memory:e7880000-e7880fff
> >>
> >>x3755b:/ # cat /etc/dat.conf
> >>ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0
> >>"eth0 0" ""
> >>
> >>Client configuration:
> >>
> >>3650m2:~ # ifconfig eth4
> >>eth4 Link encap:Ethernet HWaddr 00:07:43:07:1B:62
> >> inet addr:10.0.0.2 Bcast:10.255.255.255 Mask:255.0.0.0
> >> inet6 addr: fe80::207:43ff:fe07:1b62/64 Scope:Link
> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> >> RX packets:8730 errors:0 dropped:0 overruns:0 frame:0
> >> TX packets:8780 errors:0 dropped:0 overruns:0 carrier:0
> >> collisions:0 txqueuelen:1000
> >> RX bytes:2407318 (2.2 Mb) TX bytes:10847636 (10.3 Mb)
> >> Interrupt:26 Memory:9c800000-9c800fff
> >>
> >>
> >>3650m2:~ # cat /etc/dat.conf
> >>ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0
> >>"eth4 0" ""
> >>
> >>2. Run the below command on the server,
> >>x3755b:~ # dapltest -T S -D ofa-v2-iwarp
> >>Dapltest: Service Point Ready - ofa-v2-iwarp
> >>Dapltest: Service Point Ready - ofa-v2-iwarp
> >>
> >>3. On client, run RR dapltest as below: Note the 2 parameter for
> >>number of segments.
> >>3650m2:~ # dapltest -T P -s 10.0.0.1 -D ofa-v2-iwarp -d -i 100 RR 4096 2
> >>-------------------------------------
> >>PerfCmd.server_name : 10.0.0.1
> >>PerfCmd.dapl_name : ofa-v2-iwarp
> >>PerfCmd.mode : BLOCKING
> >>PerfCmd.num_iterations : 100
> >>PerfCmd.pipeline_len : -1
> >>PerfCmd.op.transfer_type : RDMA_READ
> >>PerfCmd.op.num_segs : 2
> >>PerfCmd.op.seg_size : 4096
> >>Server Name: 10.0.0.1
> >>Server Net Address: 10.0.0.1
> >>DT_cs_Client: Starting Test ...
> >>DT_cs_Client: IA ofa-v2-iwarp opened
> >>DT_cs_Client: EP created
> >>***** DAPL Characteristics *****
> >>Provider: ofa-v2-iwarp Version 2.0 DAPL 2.0
> >>Adapter: by Version 2.0
> >>Supporting:
> >> 32736 EPs with 1023 DTOs and 8 RDMA/RDs each
> >> 32767 EVDs of up to 65536 entries (default S/R size is 16/16)
> >> IOVs of up to 4 elements
> >> 32768 LMRs (and 0 RMRs) of up to 0xffffffff bytes
> >> Maximum MTU 0xffffffff bytes, RDMA 0xffffffff bytes
> >> Maximum Private data size 256 bytes
> >> Local IP address 10.0.0.2
> >>***** ***** ***** ***** ***** *****
> >>DT_cs_Client: Posting 1 recv buffer
> >>DT_cs_Client: Connect Endpoint
> >>DT_cs_Client: Await connection ...
> >>DT_cs_Client: Connected!
> >>DAT_STATE: DAT_EP_STATE_CONNECTED
> >>DAT_STATE: Inbound DTO Status: Idle
> >>DAT_STATE: Outbound DTO Status: Active
> >>DT_cs_Client: Sending Client_Info
> >>DT_cs_Client: Sent Client_Info - awaiting completion
> >>DT_cs_Client: Sending Command
> >>DT_cs_Client: Sent Command - awaiting completion
> >>DT_cs_Client: Waiting for Server_Info
> >>DT_cs_Client: Server_Info Received
> >>DT_cs_Client: Version OK!
> >>-------------------------------------
> >>Server_Info.dapltest_version : 6
> >>Server_Info.is_little_endian : 1
> >>-------------------------------------
> >>Client_Info.dapltest_version : 6
> >>Client_Info.is_little_endian : 1
> >>Client_Info.test_type : 2
> >>-------------------------------------
> >>PerfCmd.server_name : 10.0.0.1
> >>PerfCmd.dapl_name : ofa-v2-iwarp
> >>PerfCmd.mode : BLOCKING
> >>PerfCmd.num_iterations : 100
> >>PerfCmd.pipeline_len : -1
> >>PerfCmd.op.transfer_type : RDMA_READ
> >>PerfCmd.op.num_segs : 2
> >>PerfCmd.op.seg_size : 4096
> >>Client: Starting performance test
> >>Client[b0e0]: Connect on port 0xb0e0
> >>Client[b0e0]: Got Connection
> >>Test[b0e0]: Waiting for Sync Msg
> >>Test[b0e0]: Sync Msg Received
> >>Test[b0e0]: Begin...
> >>
> >><...test hangs...>
> >>
> >>Let me know if I can provide more information.
> >>
> >>Regards,
> >>John, SUSE
> >>_______________________________________________
> >>ewg mailing list
> >>ewg at lists.openfabrics.org
> >>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
> >
> >_______________________________________________
> >ewg mailing list
> >ewg at lists.openfabrics.org
> >http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>
>
More information about the ewg
mailing list