[Openib-windows] RE: Connection rate of WSD

[Fabian Tillier]

Hi Tzachi,

On 6/5/06, Tzachi Dar <tzachid at mellanox.co.il> wrote:
> Hi Fab,
> 1) Please see bellow my answers. I still don't see how playing with the
> rnr timeout will solve the problem.

RNR handles the case where a send is sent before a matching receive is
posted.  This is exactly the situation we're trying to handle.

Is there some problem with RNR handling in the HCAs?  The RNR
situation should not be that common, happening only durring connection
establishment on a very busy system.  The inefficiency of RNR on the
wire is worth the savings in extra complexity in the code.

I've asked about RNR handling in the HCA several times now without
getting any answer - is RNR broken in the current HCA implementations?

> 2) The way we see it there are two possible answers. A - play with the
> cm . This will slow connection establishment, but gives some more
> freedom (the CM is in software). Please also note that as for timeouts
> the CM has another message MRA (more processing required) which gives us
> exactly the freedom to do what we want. We answer that we received the
> request and still thinking what to do. So this is a timeout free
> solution.
> As for the other solution: posting the first receive this has the
> advantage that we follow the WSD spec. As for the latency introduced: I
> believe that we can add another variable that will tell if the first
> buffer was already received correctly. On the buffer complete side the
> first action will be to check if the first receive was already handled.
> Only if not, it will take the lock and do the complex thing. As a result
> I believe that the latency introduced on most of the buffers will only
> be an if statement latency, which is quiet small.
>
> What do you think?

I would rather not add code if we can avoid it.  RNR should work for
this, unless RNR is broken in the HCAs.

Maybe I don't understand - you say that a 40ms RNR retry is too long,
yet you follow with saying that it may be seconds before the switch
posts its receive.  If it's seconds, the RNR retry should be just
fine.

Anyhow, as you pointed out, sending an MRA doesn't help the connection
rate, so it's not really any better than using RNR.  I'm still
considering the buffering thing, but need to find a solution that will
be streamlined and clean.  I'm weary of making significant changes at
this point, since the RNR solution is functional.

Did you test with the updated code to see if the connection delay is
reduced?  What was the outcome?

Thanks,

- Fab