[ofa-general] librdmacm and libmthca question

[Doug Ledford]

On Thu, 2007-10-11 at 08:25 +0200, Dotan Barak wrote:
> Hi.
> 
> I can try to answer some of the questions that you have which are 
> related to the core/verbs.

> > 1) Can this limit be adjusted by module parameters, and if so, which
> > ones?
> >   
> This value is an attribute of the device (there is an upper limit on how 
> many outstanding RDMA Reads/atomic
> it supports).
> The mthca low level driver is being loaded with default value of 4 
> (which is less that the device capability),
>  but there is a module parameter called (rdb_per_qp) which can be 
> changed to support higher value.

Thanks.  I thought the rdb_per_qp might be related, but I wasn't sure.

> > 2) Does this limit represent the limit on outstanding RMDA READ/Atomic
> > operations in a) progress, b) queue, or c) registration?
> >   
> This value limit the number of RDMA read/Atomic which can be processed 
> in parallel in this QP.
> for example: you posted 100 RDMA Reads, and the QP was configured to 
> support only 4,
> so 4 RDMA Reads will be processed every time in parallel, when one will 
> be finished, another one
> will begin until all of your 100 will be processed: so the answer is a), 
> in progress.

Cool.  Then it's not really all that much of a limit in terms of my
usage anyway.  Having 4 running in parallel should be plenty to keep the
wire busy.

> > 3) The answer to #2 implies the answer to this, but I would like a
> > specific response.  If I attempt to register more IBV_ACCESS_REMOTE_READ
> > memory regions than responder resources, what happens?  If I attempt to
> > queue more IBV_WR_RDMA_READ work requests than initiator_depth, what
> > happens?  If there are more IBV_WR_RDMA_READ requests in queue than
> > initiator_depth and it hits the initiator_depth + 1 request while still
> > processing the proceeding requests, what happens?
> >   
> There isn't any connection between the number of Memory Regions that you 
> have (it doesn't matter
> which permission you registered them with) and the value that you gave 
> to the QP to handle RDMA Reads/
> Atomic. (A MR can be shared with several QPs)
> 
> I hope that i helped you with this info

Yep, very much so.

-- 
Doug Ledford <dledford at redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20071011/d0149a13/attachment.sig>