[ofiwg] send/recv "credits"

[Jason Gunthorpe]

On Sat, Sep 27, 2014 at 04:59:07PM +0000, Hefty, Sean wrote:
> </on soapbox> Honestly, I think an app designed for EAGAIN will be
> the best performing.  A credit based scheme introduces more
> branches, additional memory reads and writes, and arithmetic
> operations.  EAGAIN optimizes for a common case of having available
> queue space.  </off soapbox>

Every app I've built uses a single arithmetic to combine the sendq,
completionq, and remote recvq limit tests:

 if (have_send < can_send) // on a ring

and can_send is updated during completion processing.

AFAICS you cannot eliminate the if, except possibly, in special cases
like non-event-driven applications, and applications relying on RNR
Acks. Otherwise you always need a test of some sort.

I also tend to supress interrupts on all send completions except ones
that are the last-to-insert as an interrupt mitigation strategy. Not
sure how you do that with EAGAIN.

Also, what are the semantics of a full completion Q? Does iwarp blow
up if you have a recv wqe but no space for a cqe?

I've always designed to avoid full completion q conditions - sendq
credit is based on when the sqe is recycled back via the completion q,
and recvq credit is computed considering available space in the compq
after considering sendq usage.

Jason