[Fwd: [ofa-general] More responder_resources problems]

[Sean Hefty]

>Just wanted to be sure you saw this posting from Jason :-) If you
>haven't had time to get to it, that's fine but wanted to make sure it
>didn't get lost in the email as I've seen messages dropped... Sorry for
>the noise.

Thanks - I never saw it.

>So my expectation on how the spec outlines this should work is that
>the requesting side does essentially:
>    ibv_query_device(verbs,&devAttr);
>    req.responder_resources = devAttr.max_qp_rd_atom;
>    req.initiator_depth = devAttr.max_qp_init_rd_atom;
>
>When making the req (assuming it wants the maximum).
>
>The passive side should then take req.initiator_depth, limit it to its
>devAttr.max_qp_rd_atom (and layer a client limit on top of that) and
>assign it to max_dest_rd_atomic on its QP, and also assign it to
>rep.responder_resources.
>
>Next, the passive side should take req.responder_resources, limit it
>to devAttr.max_qp_init_rd_atom (and again layer a client limit on top of
>that), and assign it to max_rd_atomic on its QP, and return it in
>rep.initiator_depth.
>
>The active side should, generally, use the form above and use the
>values in the rep to program its max_rd_atomic and max_dest_rd_atomic.
>
>I can't find any of this in any of the cm libraries - and this is the
>sort of thing I was expecting to find in kernel cm.c, since other than
>letting the client on the passive side specify lower limits there
>really isn't much latitude here.

The initiator_depth and responder_resources are control by the CM ULP, and are
specified when calling send_cm_req / send_cm_rep.  The exchanged values are
reported through the req_event/rep_event parameters.

The behavior that you're describing is done by the kernel cm.  Look in
ib_send_cm_req / ib_send_cm_rep / cm_req_handler / cm_rep_handler.

>The particular change you introduced to support DAPL strikes me as
>just strange, overriding the incoming initator_depth with the passive
>side's responder_resources choice and then not returing that change in
>the rep makes no sense to me at all and could cause a slow down since
>the two ends are now mismatched.

The active side initiator_depth and responder_resources are set by the active
side when calling ib_send_cm_req.  The passive side initializes its values to
the data carried in the REQ.  When the passive sides sends a REP, it is allowed
to reduce the values.  The CM adjusts both the passive and active side values
based on the data in the REP.

Mismatched ends end up with the connection being broken.

>(Assuming that max_dest_rd_atomic corrisponds to responder resources
>and that max_rd_atomic corrisponds to initiator depth as discussed in

This is correct.

- Sean