[openib-general] Re: RFC: e2e credits

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
>> Hello!
>> As was recently discussed on this list, infiniband implements an
>> optional hardware end to end credits mechanism, with credits encoded
>> in the AETH field in an ACK packet.
>> 
>> The result is that an application might see performance degradation
>> even if it always posts a receive work request before remote side
>> posts a send work request, unless it keeps an extra receive
> work request available at all times.
>> Note also that most ULPs do not benefit from this mechanism because
>> they all seem to implement a software based end to end flow control.
>> 
>> A sender HCA that gets a valid credits coe in the AETH field must
>> implement the e2e flow control mechanism.
>> Note however, that it is always legal for the receiver side to
>> disable this mechanism since the spec says:
>> 
>>> 9.7.7.2.4 REQUESTER BEHAVIOR
>>> 
>>> Even a responder which does generate end-to-end credits may choose
>>> to send the 'invalid' code in the AETH.
>> 
>> At least mellanox HCAs have a per-QP option to do exactly that: send
>> an invalid credits code in AETH thus disabling e2e flow control for
>> this half connection. 
>> 
>> I am, therefore, considering one of the following options:
>> - Add a flag rq_e2e_flow_control to create qp, make e2e flow control
>> disabled by default 
>> - Add a flag disable_rq_e2e_flow_control to create qp, make e2e flwo
>> control   enabled by default 
>> - Add a module option to mthca to enable e2e flow control for all
>> QPs,   disabled by default 
>> - Add a module option to mthca to disable e2e flow control for all
>> QPs,   enabled by default 
>> - Do nothing, assume an application always keeps an extra receive
>> qork request   available at all times. 
>> 
>> Ideas?
> 
> So, how about we add a flag to ib_qp_init_attr?
> Something like
> 	responder_invalid_credits
> 
> Which will, for RC QPs, cause the responder to always
> generate invalid credits, effectively disabling hardware E2E
> flow control for this receive queue.  ULPs will turn this on
> if they do implement flow control in software and make sure that RNR
> is never generated. 
> 
> Sean?

With a properly chosen label this also documents a transport variation
in a way that is helpful to application developers. Basically when
this flag is set there is no hardware-layer credit-based flow control.
So an iWARP device would *always* have this option set.

A name like "no_hw_credit_flow_control" states the meaning to the
application in a transport neutral way.

An application could simply assume there is no hardware flow control,
and do its own flow control instead, or it could rely on hardware
flow control *if* the device attribute indicated it was enabled.