[openib-general] Lustre over OpenIB Gen2

[Sean Hefty]

Eric Barton wrote:
> 5. Should I pre-map all physical memory and do RDMA in page-sized fragments?
>    This avoids any mapping overhead at the expense of having much larger
>    numbers of queued RDMAs.  Since I try to keep up to 8 (by default) 1MByte
>    RDMAs active concurrently to any individual peer, with 4k pages I can have
>    up to 2048 RDMA work items queued at a time per peer.

This is 20 million outstanding RDMA work requests per node.

>    And if I pre-map, can I be guaranteed that if I put the CQ into the error
>    state, all remote access to my memory is revoked (e.g. could a CQ I create
>    after I destroy the one I just shut down somehow alias with it such that a
>    pathalogically delayed RDMA could write my memory)?

I think that you mean QP into the error state.  If the QP is in the error state, 
then further access from a remote system should be impossible.

- Sean