[ofiwg] send/recv "credits"

[Reese Faucette (rfaucett)]

> > Providing a mechanism for the app to efficiently query available
> > credits is perhaps the easiest, along with the "max credits" a send/recv can
> consume.
> > Reserving max_credits is much more palatable than dividing by
> > max_credits, and that would work for us.  This could even be collapsed
> > into something like "fi_ep_ok_to_send(num_sends)" which returns true
> > if the EP has space to post num_sends maximally sized sends.
> 
> How would this usage model differ significantly from the data transfer call
> simply returning FI_EGAIN?

The difference is in app usage model - typical is:
If (credits_available) {
  Do_prep_work();
  Post_send();
} else {
  Queue_for_later();
}

With the EAGAIN model, this has to change to the potentially less efficient :
Do_prep_work();
If (post_send() != OK) {
  Undo_prep_work(); 
  Queue_for_later();
}

I agree with what Doug said in Issue #10, " When it comes to applications that want to manage their credits, if we
ever return EBUSY, we've failed."

> My concern is that these methods may be restricting the underlying
> implementation, with an assumption that there is a single outbound queue.
> Based on the data ordering constraints specified by the application,
> multiple command queues could be in use.  Examples: RMA and messages
> transfers may be separated.  Message transfers may be separated based on
> transfer size.  Transfers may be separated based on the target address.  Etc.
> DAPL has code similar to this to handle performance limitations on Xeon
> Phi, and it's easier to have a layered provider within libfabric expand the
> capabilities of a lower-level provider, than have all apps implement this
> functionality or re-code to another interface.

Are you pointing out that there may be conditions where, for example "it's OK to send to X, but sending to Y would need to return EBUSY" ?  More generally, that the ability of an EP to accept a particular send cannot be determined by a simple scalar compare ?  OK, I buy that.  Suppose there's an ep_attr FI_CREDIT_BASED that indicates whether a scalar credit mechanism can be used, or whether the app needs to fall back to attempting the post and look for EAGAIN ? 

Even in the non-credit-based approached, one could still possibly do a pre-check by having essentially a "send_peek" - "If I try this send, will it succeed or not?"  I can't really come up with a great use-case for that, just tossing it out there.  Marshalling the "send_peek" call is probably not any less work for the app than trying the actual send.

-reese