[openib-general] Re: RFC: e2e credits

[Michael S. Tsirkin]

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Hello!
> As was recently discussed on this list, infiniband implements
> an optional hardware end to end credits mechanism, with credits
> encoded in the AETH field in an ACK packet.
> 
> The result is that an application might see performance degradation even if it
> always posts a receive work request before remote side posts a send work
> request, unless it keeps an extra receive work request available at all times.
> Note also that most ULPs do not benefit from this mechanism because they all
> seem to implement a software based end to end flow control.
> 
> A sender HCA that gets a valid credits coe in the AETH field must
> implement the e2e flow control mechanism.
> Note however, that it is always legal for the receiver side to disable
> this mechanism since the spec says:
> 
> > 9.7.7.2.4 REQUESTER BEHAVIOR
> > 
> > Even a responder which does generate end-to-end credits may
> > choose to send the 'invalid' code in the AETH.
> 
> At least mellanox HCAs have a per-QP option to do exactly that: send
> an invalid credits code in AETH thus disabling e2e flow control for this
> half connection.
> 
> I am, therefore, considering one of the following options:
> - Add a flag rq_e2e_flow_control to create qp, make e2e flow control
>   disabled by default
> - Add a flag disable_rq_e2e_flow_control to create qp, make e2e flwo control
>   enabled by default
> - Add a module option to mthca to enable e2e flow control for all QPs,
>   disabled by default
> - Add a module option to mthca to disable e2e flow control for all QPs,
>   enabled by default
> - Do nothing, assume an application always keeps an extra receive qork request
>   available at all times.
> 
> Ideas?

So, how about we add a flag to ib_qp_init_attr?
Something like
	responder_invalid_credits

Which will, for RC QPs, cause the responder to always generate invalid credits,
effectively disabling hardware E2E flow control for this receive queue.  ULPs
will turn this on if they do implement flow control in software and make sure
that RNR is never generated.

Sean?

-- 
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies