[ofiwg] send/recv "credits"

[Hefty, Sean]

Thinking about this more, there's still an underlying assumption that there's some sort of fixed size queue with a well-defined relation to a WR.  IB defines a 1:1 mapping between a posted WR and a queue entry.  Cisco defines a 1:1 mapping between an SGE and a queue entry.

Conceptually, if I think of this in terms of CPU instructions (since I work for Intel :), we can have fixed sized instructions (IB-like case) or variable sized instructions (which is closer to the Cisco model).

If we don't impose any restrictions on the implementation, another possibility is that the queuing model may be related to data buffering, rather than WRs, or even a combination of both.  The current concept of inline_data highlights this.  In the IB model, inline_data replaces the WR.  But if the Cisco model added support for inline_data, then there's not a well-defined relationship between the WR and the amount of queue space consumed.  Inline_data may be injected directly into the command queue, but we're dealing with a variable sized command.

A credit based model may arbitrarily limit possible implementations, unless there's a way to indicate that the provider cannot support that model.  And if that's the case, then apps would need to code for the non-credit scheme anyway.  Even providing a call, can_I_post_a_send_now(), may be of little use, since the answer may ultimately depend on knowing the full details of the WR to post.  This is similar to the socket select/poll call, which only indicates that the socket is ready to accept _some_ amount of data.

On the receive side, we may have a similar issue.  A sender may be blocked not because of how many receives have been posted, but how much buffer space is available, or even both.

- Sean