[ofiwg] send/recv "credits"

[Hefty, Sean]

> Well, I don't see how you can expect an application to efficiently
> manage message flow to hidden queues. At a minimum an app needs to
> know messages can be submitted, know when the hidden queue is full and
> be able to sleep until it becomes !full.

A viable solution is to require providers to support an infinite queue, subject to memory constraints.  I don't think MPI limits what an application can queue.  I doubt everyone will be happy with the potential performance impact of this, but then again, the request is going to be queued somewhere...

> Once you provide APIs for all that, the queue is no longer hidden, the
> abstraction is broken - so why use it at all?

Separate from this, I'm trying to come up with a way to expose the case where multiple command queues may be associated with a single endpoint.  Once that's defined, exposing queue sizes may be relatively straightforward.

Then all that will be left is exposing limits on send side buffering, which may be independent of queue space, completion restrictions, etc. so that a send call will actually proceed.

I'm actually more concerned about credits at the receive side.  :)