[openib-general] RFC: e2e credits

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> Hello!
> As was recently discussed on this list, infiniband implements
> an optional hardware end to end credits mechanism, with
> credits encoded in the AETH field in an ACK packet.
> 
> The result is that an application might see performance
> degradation even if it always posts a receive work request
> before remote side posts a send work request, unless it keeps
> an extra receive work request available at all times.
> Note also that most ULPs do not benefit from this mechanism
> because they all seem to implement a software based end to
> end flow control.
> 
> A sender HCA that gets a valid credits coe in the AETH field
> must implement the e2e flow control mechanism.
> Note however, that it is always legal for the receiver side
> to disable this mechanism since the spec says:
> 
>> 9.7.7.2.4 REQUESTER BEHAVIOR
>> 
>> Even a responder which does generate end-to-end credits may choose to
>> send the 'invalid' code in the AETH.
> 
> At least mellanox HCAs have a per-QP option to do exactly
> that: send an invalid credits code in AETH thus disabling e2e
> flow control for this half connection.
> 
> I am, therefore, considering one of the following options:
> - Add a flag rq_e2e_flow_control to create qp, make e2e flow control 
> disabled by default 
> - Add a flag disable_rq_e2e_flow_control to create qp, make
> e2e flwo control
>   enabled by default
> - Add a module option to mthca to enable e2e flow control for all
> QPs,   disabled by default 
> - Add a module option to mthca to disable e2e flow control
> for all QPs,
>   enabled by default
> - Do nothing, assume an application always keeps an extra
> receive qork request
>   available at all times.
> 
> Ideas?


Transport neutral applications MUST assume that they are responsible
for flow control of messages that cause remote completions.

So what you are really talking about is not disabling end to end
flow control, but disabling *hardware* enforced end to end flow control.

With that clarification, I think it is a great idea. Since I think
reliance on true end to end flow control (i.e., ULP based) is how
applications SHOULD be written, I see no problem with this being
a default and/or the control being coarse rather than per QP.

Does anyone have a scenario where an application would rely on
hardware based end to end credits for some QPs and not others
running in the same application?