[ofiwg] send/recv "credits"

[Jason Gunthorpe]

On Wed, Sep 24, 2014 at 07:36:19PM +0000, Hefty, Sean wrote:
> > Providing a mechanism for the app to efficiently query available credits is
> > perhaps the easiest, along with the "max credits" a send/recv can consume.
> > Reserving max_credits is much more palatable than dividing by max_credits,
> > and that would work for us.  This could even be collapsed into something
> > like "fi_ep_ok_to_send(num_sends)" which returns true if the EP has space
> > to post num_sends maximally sized sends.
> 
> How would this usage model differ significantly from the data
> transfer call simply returning FI_EGAIN?

Conceptually, I agree the actual number of entires is not very
important if a post can fail with 'full', but in a practical sense I
think it is necessary for an app to be able to query if a queue is
full before it commits to producing a message, otherwise it has to
have code to unwind a message create because post had a temporary
failure.

An API that provided a guaranteed count of entires that could be added
right now would be efficient and simple - posting and repeatedly
calling when the count reaches 0 would allow any single queue
construction to become full:

while (1)
{
   unsigned int avail = fi_get_avail_posts(..);
   if (avail == 0)
      break;
   for (; avail != 0; avail--)
      fi_post(...)
}

Underlying models that do not have a 1:1 correspondance of HW resource
and post action will have increased overhead, but still work
correctly.

> My concern is that these methods may be restricting the underlying
> implementation, with an assumption that there is a single outbound
> queue.  Based on the data ordering constraints specified by the
> application, multiple command queues could be in use.  Examples: RMA
> and messages transfers may be separated.  Message transfers may be
> separated based on transfer size.  Transfers may be separated based
> on the target address.  

Well, I don't see how you can expect an application to efficiently
manage message flow to hidden queues. At a minimum an app needs to
know messages can be submitted, know when the hidden queue is full and
be able to sleep until it becomes !full.

Once you provide APIs for all that, the queue is no longer hidden, the
abstraction is broken - so why use it at all?

The simplest API is one that says once any hidden queue is full, then
everything stops, that is also the lowest performance if the goal is
to fill queues.

Jason