[ofiwg] send/recv "credits"

[Reese Faucette (rfaucett)]

> A viable solution is to require providers to support an infinite queue,
> subject to memory constraints.  I don't think MPI limits what an application
> can queue.  I doubt everyone will be happy with the potential performance
> impact of this, but then again, the request is going to be queued
> somewhere...

I think instead of infinite it would need to be some user-adjustable equivalent of SO_SENDBUF that was query-able as to how full it is.  A nice feature of this queue is that it could be in units of "send requests" as opposed to more HW-specific  "queue entries."  I'll think more about this, but the two issues that spring to mind are:

	- performance - providers fight for nanoseconds and giving them up to API semantics is a bitter pill.  I doubt usnic and Infiniband are unique in benefitting from avoiding every possible conditional in the send fastpath.

	- progression - normally, at least for UD, once something is "posted", it will go out on the wire with no further progression (either AUTO or MANUAL).   Maybe datagram EPs are already allowed to require progression, but this queueing would definitely require progression.
 
> I'm actually more concerned about credits at the receive side.  :)

Yes, IMO this whole thread applies symmetrically to posting receives.

-reese