[openib-general] RFC: e2e credits

[Michael S. Tsirkin]

Hello!
As was recently discussed on this list, infiniband implements
an optional hardware end to end credits mechanism, with credits
encoded in the AETH field in an ACK packet.

The result is that an application might see performance degradation even if it
always posts a receive work request before remote side posts a send work
request, unless it keeps an extra receive work request available at all times.
Note also that most ULPs do not benefit from this mechanism because they all
seem to implement a software based end to end flow control.

A sender HCA that gets a valid credits coe in the AETH field must
implement the e2e flow control mechanism.
Note however, that it is always legal for the receiver side to disable
this mechanism since the spec says:

> 9.7.7.2.4 REQUESTER BEHAVIOR
> 
> Even a responder which does generate end-to-end credits may
> choose to send the 'invalid' code in the AETH.

At least mellanox HCAs have a per-QP option to do exactly that: send
an invalid credits code in AETH thus disabling e2e flow control for this
half connection.

I am, therefore, considering one of the following options:
- Add a flag rq_e2e_flow_control to create qp, make e2e flow control
  disabled by default
- Add a flag disable_rq_e2e_flow_control to create qp, make e2e flwo control
  enabled by default
- Add a module option to mthca to enable e2e flow control for all QPs,
  disabled by default
- Add a module option to mthca to disable e2e flow control for all QPs,
  enabled by default
- Do nothing, assume an application always keeps an extra receive qork request
  available at all times.

Ideas?

-- 
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies