[ofiwg] send/recv "credits"

[Hefty, Sean]

> > Independent from EAGAIN, Does the op_size / iov_size / op_alignment
> > proposal work for apps that want to track send queue usage separate
> > from the provider's tracking?
> 
> I didn't follow it too closely, sorry.  How does an app adapt a
> provider that is telling it to use sge entries to work with a wire
> protocol that is defined in terms of wqes?

The size of the transmit queue is reported in bytes.  An app does this check to determine if it can queue an entry into the transmit queue.  (An app can simplify this check in certain cases.)

	needed = ((op_size + iov_size * nsge) + op_alignment - 1) & ~(op_alignment - 1)

For providers that support WQEs, needed = op_alignment.
For providers that support SGEs, needed = iov_size * nsge.

This should also support providers where the size of the queue is fixed, but the number of entries is not.

> The remote CQ doesn't overflow because every SQE and RQE is still
> guarenteed by the app to have an available CQE before it is posted. So
> you are guarenteed to hit RQ exhaustion before you hit CQ exhaustion.

Libfabric supports, but does not assume a 1:1 mapping between a posted receive buffer and a CQE.  This allows for more efficient use of receive buffering, but does require a more advanced form of flow control than current hardware supports.

I don't want the API to assume that even a CQ has a fixed number of entries.  An app should be able to determine the minimum number of entries any queue may support, without restricting all providers or applications to using that same model.

- Sean