[ofa-general] RDS flow control

[Steve Wise]

Olaf Kirch wrote:
> On Monday 12 May 2008 18:57:38 Jon Mason wrote:
>   
>> As part of my effort to get RDS working for iWARP, I will be working on the 
>> RDS flow control.  Flow control is needed for iWARP due to the fact that 
>> iWARP connections terminate if there is no posted recv for an incoming 
>> packet.  IB connections do not have this limitation if setup in a certain 
>> way.  In its current implementation, RDS sets the connection attribute 
>> rnr_retry to 7.  This causes IB to retransmit until there is a posted recv 
>> buffer. 
>>     
>
> I think for the initial implementation, it is fine for iWARP to just
> fail the connect when that happens, and re-establish the connection.
>
> If you use reasonable defaults for the send and recv queues, receiver
> overruns should be relatively rare.
>
> Once everything else works, let's revisit the flow control part.
>
>   
I _think_ you'll hit this quickly with one-way flows.  Send completions 
for iWARP only mean the user's buffer can be reused.  Not that its 
placed at the remote peer or in the remote user's buffer.

But perhaps I'm wrong.  Jon, maybe you should try to hit this with IB 
and rnr_retry == 0 using the rds perf tools? 

Also "the everything else" part depends on remove fmr usage.  I'm 
working on the new RDMA memory verbs allowing fast registration of 
physical memory via a send WR.  To support iWARP we need to remove the 
fmr usage from RDS.   The idea was to replace fmrs with the new fastreg 
verbs.   Thoughts?

Stay tuned for the new verbs API RFC...

Steve.