[ofiwg] send/recv "credits"

[Jason Gunthorpe]

On Mon, Sep 29, 2014 at 12:08:47AM +0000, Hefty, Sean wrote:

> > I've always designed to avoid full completion q conditions - sendq
> > credit is based on when the sqe is recycled back via the completion q,
> > and recvq credit is computed considering available space in the compq
> > after considering sendq usage.
> 
> But could you have designed it around EAGAIN -- assuming that you
> had real flow control support?

Then you go from 

if (have_sent < can_send) {
  post()
}

To:

if (failed_send) {
  if (post(failed_send) == EGAIN)
      return ...

if (post(cur_send) == EGAIN)
  failed_send = cur_send

And again, I don't see how that is more efficient or simpler.

I'm sort of confused here, introducing hidden flow control is a
middleware function - libfabric will need its own private frame
structure to make that work portably. Is that the intent?

> Actually, a completion queue may use variable sized entries as well.

It doesn't seem something like that could exist without a hidden
on-the-wire flow control scheme of some sort (like RNR ACK).

We already basically have that requirement for many SRQ and XRC type
applications, so it isn't really a new thing.

Jason