[ofiwg] send/recv "credits"

[Hefty, Sean]

> Also, what are the semantics of a full completion Q? Does iwarp blow
> up if you have a recv wqe but no space for a cqe?

I *think* iWarp blows up the same as IB.

Okay, I see this as a problem.  IMO, this is a violation of what an app should be able to expect from an endpoint with flow control enabled.  With flow control, the app shouldn't care what set of buffers may be limited, the provider just needs to deal with it.  Use an RNR NACK if there's no posted receive, the receive buffer is too small, there's no CQ space, whatever.

Sigh, I guess there needs to be multiple levels of flow control defined...

> I've always designed to avoid full completion q conditions - sendq
> credit is based on when the sqe is recycled back via the completion q,
> and recvq credit is computed considering available space in the compq
> after considering sendq usage.

But could you have designed it around EAGAIN -- assuming that you had real flow control support?

For reliable unconnected endpoints or endpoints that support 'slab based' receive buffering (a single buffer accepts data from multiple sends), the scheme gets more complex.  I don't want to assume anything about the implementation of the transmit queue.  It may not be a fixed number of entries, or the queuing of the request may be done entirely by software, with a progress thread responsible for processing it.

Actually, a completion queue may use variable sized entries as well.

- Sean