[ewg] RE: [ofa-general] how do I use uDAPL with iWARP?

[Steve Wise]

Guys,

I think this is the same iWARP issue that has been biting me for a 
while:  The client must send the first RDMA message.  The dtest app is a 
peer-2-peer (p2p) application where both sides send immediately after 
setting up the connection.  So dtest doesn't adhere to the iWARP 
specification (I know: the iWARP spec is broken :).

News:  I have some prototype FW from chelsio that supports p2p setup and 
with that FW and my associated iw_cxgb3 driver/library changes, then 
dtest seems to work fine.  These changes will be published upstream soon 
in order to support Open MPI and other p2p applications for chelsio.

For this initial release of p2p support over chelsio, the functionality 
will be 100% handled in the iw_cxgb3 driver and fw.  This is similar to 
what iw_nes does today with its send_first module option to send a 0B 
write from the client and defer connection establishment on the server 
until the 0B write is received.  Chelsio will have a similar module 
option called peer2peer (or I could make it the same option name: 
send_first) that will use a 0B read to force the client to send first 
(chelsio cannot use a 0B write for this).  The chelsio FW will defer the 
ESTABLISHED event until the 0B read is received and responded to.

The final proper device-independent solution to this will be done in the 
rdma-cma, the iwarp core and iwarp devices for upstream inclusion as 
well as for ofed-1.4.  Its a much bigger change and will affect the ABI 
for the rdma_cm probably (app can request p2p behavior).  There was a 
thread a while back driven by Arkady at NetApp with details on how we 
will implement this (using a small protocol in mpa start req/rep to 
negotiate this p2p mode).  Stay tuned for more on this.



Steve.


Steve Wise wrote:
> I can reproduce this.  Lemme dig into it...
> 
> Steve.
> 
> 
> Scott Weitzenkamp (sweitzen) wrote:
>> Steve,
>>
>> Thanks, that gets further, but dtest still fails.
>>
>> Client side:
>>
>> [releng at svbu-qa2950-2 ~]$ DAPL_MAX_INLINE=64 dtest -h 192.168.0.198
>> 13926 Running as client - OpenIB-cma
>> 13926 Server Name: 192.168.0.198
>> 13926 Server Net Address: 192.168.0.198
>> 13926 Waiting for connect response
>> 13926 Error unexpected conn event : DAT_CONNECTION_EVENT_UNREACHABLE
>> 13926 Error connect_ep: DAT_ABORT
>>
>> 13926: DAPL Test Complete.
>>
>> 13926: Message RTT: Total=      0.00 usec, 10 bursts, itime=      0.00
>> usec, pc=
>> 0
>> 13926: RDMA write:  Total=      0.00 usec, 10 bursts, itime=      0.00
>> usec, pc=
>> 0
>> 13926: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
>> usec, pc
>> =0
>> 13926: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
>> usec, pc
>> =0
>> 13926: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
>> usec, pc
>> =0
>> 13926: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
>> usec, pc
>> =0
>> 13926: open:        36619.19 usec
>> 13926: close:       32500.98 usec
>> 13926: PZ create:       7.87 usec
>> 13926: PZ free:         4.05 usec
>> 13926: LMR create:     58.89 usec
>> 13926: LMR free:       11.92 usec
>> 13926: EVD create:      9.78 usec
>> 13926: EVD free:       14.07 usec
>> 13926: EP create:      78.92 usec
>> 13926: EP free:        26.23 usec
>> 13926: TOTAL:         199.79 usec
>>
>> Server side:
>>
>> [releng at svbu-qa2950-1 ~]$ DAPL_MAX_INLINE=64 dtest
>> 11461 Running as server - OpenIB-cma
>> 11461 Server waiting for connect request..
>> 11461 Waiting for connect response
>>
>> 11461 CONNECTED!
>>
>> 11461 Send RMR to remote: snd_msg:
>> r_key_ctx=bff,pad=0,va=146db580,len=0x40
>> 11461 Waiting for remote to send RMR data
>> 11461 Error waiting on h_dto_rcv_evd: DAT_TIMEOUT_EXPIRED
>> 11461 Error connect_ep: DAT_TIMEOUT_EXPIRED
>>
>> 11461: DAPL Test Complete.
>>
>> 11461: Message RTT: Total=      0.00 usec, 10 bursts, itime=      0.00
>> usec, pc=
>> 0
>> 11461: RDMA write:  Total=      0.00 usec, 10 bursts, itime=      0.00
>> usec, pc=
>> 0
>> 11461: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
>> usec, pc
>> =0
>> 11461: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
>> usec, pc
>> =0
>> 11461: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
>> usec, pc
>> =0
>> 11461: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
>> usec, pc
>> =0
>> 11461: open:       900676.01 usec
>> 11461: close:       31543.97 usec
>> 11461: PZ create:       7.87 usec
>> 11461: PZ free:         5.01 usec
>> 11461: LMR create:     51.98 usec
>> 11461: LMR free:       12.16 usec
>> 11461: EVD create:     10.97 usec
>> 11461: EVD free:       12.87 usec
>> 11461: EP create:      77.01 usec
>> 11461: EP free:        30.04 usec
>> 11461: TOTAL:         195.03 usec
>>
>> Scott
>>
>>  
>>
>>> -----Original Message-----
>>> From: Steve Wise [mailto:swise at opengridcomputing.com] Sent: Thursday, 
>>> April 03, 2008 9:19 AM
>>> To: Scott Weitzenkamp (sweitzen)
>>> Cc: Joshua Bernstein; OpenFabrics EWG; [ofa_general]
>>> Subject: Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP?
>>>
>>>
>>>
>>> Scott Weitzenkamp (sweitzen) wrote:
>>>> I tried that, and it didn't work:
>>>>
>>>> [root at svbu-qa2950-1 ~]# grep eth /etc/dat.conf
>>>> OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 
>>> dapl.1.2 "eth2 0"
>>>> ""
>>>> [root at svbu-qa2950-1 ~]# dtest
>>>> 10194 Running as server - OpenIB-cma
>>>> 10194 Error dat_ep_create: DAT_INVALID_HANDLE
>>>> 10194 Error freeing EP: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_EP
>>>>
>>> try setting DAPL_MAX_INLINE=64
>>>
>>>
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg