[ofa-general] RDS flow control

[Steve Wise]

Richard Frank wrote:
> Steve Wise wrote:
>> Olaf Kirch wrote:
>>> On Monday 12 May 2008 18:57:38 Jon Mason wrote:
>>>  
>>>> As part of my effort to get RDS working for iWARP, I will be 
>>>> working on the RDS flow control.  Flow control is needed for iWARP 
>>>> due to the fact that iWARP connections terminate if there is no 
>>>> posted recv for an incoming packet.  IB connections do not have 
>>>> this limitation if setup in a certain way.  In its current 
>>>> implementation, RDS sets the connection attribute rnr_retry to 7.  
>>>> This causes IB to retransmit until there is a posted recv buffer.     
>>>
>>> I think for the initial implementation, it is fine for iWARP to just
>>> fail the connect when that happens, and re-establish the connection.
>>>
>>> If you use reasonable defaults for the send and recv queues, receiver
>>> overruns should be relatively rare.
>>>
>>> Once everything else works, let's revisit the flow control part.
>>>
>>>   
>> I _think_ you'll hit this quickly with one-way flows.  Send 
>> completions for iWARP only mean the user's buffer can be reused.  Not 
>> that its placed at the remote peer or in the remote user's buffer.
>>
> Let's see what happens - anyway - this could be solved in an IWARP 
> extension to RDS  - right ?


Yes, by adding flow control.  And it could be iwarp-specific if you 
want.    I would not suggest relying on connection termination and 
re-establishment as the way to handle this :).



>> But perhaps I'm wrong.  Jon, maybe you should try to hit this with IB 
>> and rnr_retry == 0 using the rds perf tools?
>> Also "the everything else" part depends on remove fmr usage.  I'm 
>> working on the new RDMA memory verbs allowing fast registration of 
>> physical memory via a send WR.  To support iWARP we need to remove 
>> the fmr usage from RDS.   The idea was to replace fmrs with the new 
>> fastreg verbs.   Thoughts?
>>
> What does "fast" imply here - how does this compare to the performance 
> of FMRs ?


Don't know yet, but probably as fast. 

>
> Why would not push memory window creation into the RDS transport 
> specific implementations ?

Isn't it already transport-specific?  IE you don't need FMRs for TCP.  
(I'm ignorant on the specifics of the implementation at this point, so 
please excuse any dumb statements :)


>
> Changing the API may be OK - if we retain the performance we have with 
> IB.


I assume nothing would fly that regresses IB performance.  Worst case, 
you have an iwarp-specific RDS transport like you do for TCP, I guess.  
Hopefully though, IB + iWARP will be a common transport.


>
>> Stay tuned for the new verbs API RFC...
>>
>> Steve.
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit 
>> http://openib.org/mailman/listinfo/openib-general