[ofa-general] Can you clarify my 'RNR NAK timer' understanding ?

[Todd Rimmer]

> From: Tang, Changqing
> Sent: Thursday, September 20, 2007 12:48 PM
> To: Michael S. Tsirkin
> Cc: general at lists.openfabrics.org
> Subject: [ofa-general] Can you clarify my 'RNR NAK timer'
understanding ?
> 
> 
> If I set RNR NAK timer to the biggest value (00000, 655.38 ms), when
the
> HCA recevies
> a message and no receive WR outstanding,  it will wait 655.38 ms, then
> it sends a RNR
> NAK back.
> 
> If I post a receive WR during this waiting time, then the message will
> be received, and
> RNR NAK won't be sent.
> 
> Am I right ?   What is the side effect if I set RNR NAK timer a big
> timer ?

No, the behavior is if an input message arrives at a QP without any
receive Q entries, the receiver immediately sends an RNR NAK.  The RNR
NAK timeout is part of the RNR NAK packet.

In response to the RNR NAK packet, the sender waits at least the given
timeout before retrying the send (actual wait time could be more than
the requested value).

Hence supplying a large RNR NAK timeout means there will be a large
penalty when you run out of receive buffers.  In this case your
application will "stall" for at least the RNR NAK timeout duration even
if the receiver replenishes its queue shortly after the RNR NAK is sent.

One alternative is a smaller RNR NAK timeout on the receiver and a large
(or infinite) RNR Retry on the sender.  This avoids the time delay
penalty, however it wastes bandwidth in the fabric if the sender resends
before the receiver has replenished its receiver queue.

The best alternative, which is employed by many IB protocols such as
SRP, SDP, etc is to track application level credits so the sender will
not attempt to send into an empty receive Q.  Obviously that requires
changes to the application protocol being designed.

The tradeoff of performance vs wasting bandwidth vs protocol complexity
will depend on the specific application/problem you are trying to solve.

Todd Rimmer