[ofa-general] librdmacm and libmthca question

[Doug Ledford]

OK, I ran into an issue with librdmacm and I was curious what the
answers to these issues are.

First, the rdma_connect/rdma_accept functions both require a connection
param struct.  That struct tells librdmacm what you want in terms of
responder_resources and initiator_depth.  Reading the man page, that's
the number of outstanding RMDA reads and RDMA atomic operations.  In
usage, I found that the QP max_recv_wr and max_send_wr are totally
unrelated to this (I at first thought they could be the same).  In fact,
on mthca hardware I found the hard limit to be either 4 or 5 (4 worked,
6 didn't, didn't try 5, assumed 4).  So even with a send queue depth of
128, I couldn't get above a 4 depth on initiator_depth.  I think it
might be of value to document somewhere that the initiator depth and
responder resources are not directly related to the actual work queue
depth, and that without some sort of intervention, are not that high.

However, I spent a *lot* of time tracking this down because the failure
doesn't occur until rdma_accept time.  Passing an impossibly high value
in initiator_depth or responder_resources doesn't fail on rdma_connect.
This leads one to believe that the values are OK, even though they fail
when you use the same values in rdma_accept.  A note to this effect in
the man pages would help.

Second, now that I know that mthca hardware fails with initiator depth
or responder resources > 4, it raises several unanswered questions:

1) Can this limit be adjusted by module parameters, and if so, which
ones?

2) Does this limit represent the limit on outstanding RMDA READ/Atomic
operations in a) progress, b) queue, or c) registration?

3) The answer to #2 implies the answer to this, but I would like a
specific response.  If I attempt to register more IBV_ACCESS_REMOTE_READ
memory regions than responder resources, what happens?  If I attempt to
queue more IBV_WR_RDMA_READ work requests than initiator_depth, what
happens?  If there are more IBV_WR_RDMA_READ requests in queue than
initiator_depth and it hits the initiator_depth + 1 request while still
processing the proceeding requests, what happens?

-- 
Doug Ledford <dledford at redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20071011/c4a866e4/attachment.sig>