[ofa-general] rdma_create_qp fails with -12

[Shipman, Galen M.]

> physically contiguous).  Removing this limitation would make the code
> more complex, and in general supporting huge queue depths hasn't
> seemed that important.

Currently lnet 1.4.11 (lustre networking layer) uses 4K work requests  
(page sizes).
Up to 256 of these are chained together using the next pointer so  
that a single call to ib_post_send is made for up to a 1MB xfer.
The number of work requests allocated for the QP is controlled by  
number of concurrent sends * 256.

At 16 concurrent sends there is no problem.
At 64 there is (once we allocate recv work requests as well).

It sounds like this can be alleviated by using FMR.

- Galen