[Fwd: [ofa-general] More responder_resources problems]

[Sean Hefty]

>Ok.. Well, if the ULP is responsible, I have yet to see a ULP example,
>or in-kernel ULP that does it right. Every one ignores the REQ and/or does
>not limit the REQ's values to the devices capabilities.

I believe that DAPL does negotiate the values correctly.  But see the end of
this email for a way to simply things for the ULPs.

>But the whole point of this process is to get a working connection -
>the responder resources are not a ULP visible item, they are just
>something that must be negotiated and configured into the QP. In
>truth, I can think of no reason for a ULP to use any value other than
>the device maximum or 0 for these resources. Saying that if the
>passive side messes up it will just die when the QP is modified is,
>IMHO, not good enough.

For the IB CM, the policy controlling the use of those fields is given to the
ULP.  A check could be added to ib_send_cm_rep to fail if the ULP tries to use a
value higher than that in the REQ.  I would not have the CM automatically
replace the user's values with its own.

For the RDMA CM, there's no guarantee that the initiator_depth and
responder_resources are available in the connection request.  With iWarp, the
values are not available unless embedded somewhere in the private data.
 
>That is not entirely true, since the passive side's change overrides
>the values in the REQ from the active side, which can reduce the value
>without the users knowledge. The question really is if you expect the
>CM to control this for you, or if you expect the ULP do do everything
>manually. Right now there seems to be a bit of both going on.

The values in the REP are set by one user and given to the other.  Just because
the ULP ignores the value doesn't mean that it's hidden.  The ULP really should
control the policy on how to respond to a REQ or REP based on the values that
are carried.

>> >Also support user passive side control over initiator depth.
>>
>> This is there today.
>
>Where? cma.c never programs max_rd_atomic in the qp.

rdma_accept() takes the responder_resources and initiator_depth as part of its
input parameter.  These are passed to the CM, which end up being used when
getting the modify QP attributes.

>Well, what I have been interested in (Hal - what is your interest
>here?) is to use the device maximum and get rid of the hard coded
>values for responder resources and initiator depth in the ULPs. This
>would be to allow some devices to have higher responder resources,
>based on hardware capabilitity. Limited responder resources cause huge
>performance problems on high latency connections.

To make it easier on the active side, we could allow the user to specify some
'MAX_RDMA' value that either the rdma cm or ib cm can key off of.  The cm could
then request initiator_depth and responder_resources based on the local HW
maximums.  The passive side could also specify MAX_RDMA, which for IB would
negotiate down to the values in the REQ and the local HW resources.

This doesn't really work for iWarp, but then unless the data is exchanged as
part of the private data, the best that the cm could do is guess based on the
local HW maximums.  In practice, this would probably work the majority of the
time though.

- Sean