[ofa-general] More responder_resources problems

[Jason Gunthorpe]

Hey Sean,

I was just looking at tuning the responder_resources and I'm not quite
sure what the intent of your implementation is regarding this.. Right
now I'm mostly looking at userspace through libibcm, but in-kernel and
librdmacm seem to have similar issues. I suppose this is related to
your recent changesets 3eb99a28f41392f8555977aa12a345d251d218b3
(librdmacm) and 5851bb893e5bb87150817c180ccddcf4e78db1b6 (kernel)..

Basically, it seems to me that the negotiation protocol for
responder_resources/initiator_depth that is envisioned in IBA is not
implemented..

So my expectation on how the spec outlines this should work is that
the requesting side does essentially:
    ibv_query_device(verbs,&devAttr);
    req.responder_resources = devAttr.max_qp_rd_atom;
    req.initiator_depth = devAttr.max_qp_init_rd_atom;

When making the req (assuming it wants the maximum).

The passive side should then take req.initiator_depth, limit it to its
devAttr.max_qp_rd_atom (and layer a client limit on top of that) and
assign it to max_dest_rd_atomic on its QP, and also assign it to
rep.responder_resources.

Next, the passive side should take req.responder_resources, limit it
to devAttr.max_qp_init_rd_atom (and again layer a client limit on top of
that), and assign it to max_rd_atomic on its QP, and return it in
rep.initiator_depth.

The active side should, generally, use the form above and use the
values in the rep to program its max_rd_atomic and max_dest_rd_atomic.

I can't find any of this in any of the cm libraries - and this is the
sort of thing I was expecting to find in kernel cm.c, since other than
letting the client on the passive side specify lower limits there
really isn't much latitude here.

The particular change you introduced to support DAPL strikes me as
just strange, overriding the incoming initator_depth with the passive
side's responder_resources choice and then not returing that change in
the rep makes no sense to me at all and could cause a slow down since
the two ends are now mismatched.

(Assuming that max_dest_rd_atomic corrisponds to responder resources
and that max_rd_atomic corrisponds to initiator depth as discussed in
11.2.4.3, Dotan: It should would be nice if ibv_modify_qp(3) used
the terms from IBA to describe them ..)

What do you think?

Thanks,
-- 
Jason Gunthorpe <jgunthorpe at obsidianresearch.com>        (780)4406067x832
Chief Technology Officer, Obsidian Research Corp         Edmonton, Canada