[openib-general] RE: Re: RFC: e2e credits

[Caitlin Bestler]

Sean Hefty wrote:
>> What does the flow_control parameter in ib_cm and rdma_cma
>> conn_param do? If that's what it is, I think its easier to just add
>> API to ib verbs making it possible to implement it than invent a
>> device-specific way to do it per driver.
> 
> As far as I can tell, it doesn't do anything.  Flow control
> is a parameter of the IB CM REQ/REP messages, so are exposed to the
> user. 
> 
> I don't see how a user can make use of it in any practical
> way, however.  It isn't used to configure anything on the QP
> from what I can tell.  This is why I mentioned that the only
> use I can think of is for an app to change its algorithms
> based on whether this value is set or not, but I have a hard
> time imagining an app doing so.
> 
> - Sean

The approach taken in both DAT and IT-API is to provide the
flow control information to both the Connection Manager *and*
in the private data. IT-API even defines a standard pre-header
for the Private Data over iWARP to carry this information.

When the transport specific Connection Manager can use this data
it does so directly. Otherwise the peer uses the information in
the Private Data to configure/select its QP to match. With IT-API
this is handled by middleware, while with DAT the application
itself is expected to do this.

An application can maintain transport neutrality by simply complying
with the flow control limit at all times (in selection/configuration
of the Endpoint and in how many unacknowledged Sends it generates)
whether or not there is any hardware enforcement.

The type of application that can use hardware flow control is
indeed relatively rare. It is one that has no real need for
explicit positive acknowledgement, and instead infers acceptance
from a transport layer ack and a lack of a complaint/nack.

Those applications *could* make use of such an indicator, by
explicitly generating periodic acks in the absence of hardware
flow control. Whether checking the flag is worthwhile, as opposed
to always using explicit acks, would depend on the application.
I would only rely on hardware flow control when my application
had very specific needs that were not addressed by the more
transport neutral logic.

But as a very layer interface (as opposed to an application
layer interface such as DAPL) making presumptions about what
an application will find useful is something that should be
kept to a minimum.