[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Or Gerlitz]

Steve Wise wrote:
> Support for the IB BMME and iWARP equivalent memory extensions to 
> non shared memory regions.  Usage Model:
> - MR allocated with ib_alloc_mr()
> - Page lists allocated via ib_alloc_fast_reg_page_list().
> - MR made VALID and bound to a specific page list via ib_post_send(IB_WR_FAST_REG_MR)
> - MR made INVALID via ib_post_send(IB_WR_INVALIDATE_MR)
Steve,

I am trying to further understand what would be a real life ULP design 
here, and I think there are some more issues to clarify/define for the 
case of ULP which has to create a mapping for a list of pages and send 
this mapping (eg IB/rkey iWARP/stag) to a remote party that uses it for 
RDMA.

AFAIK, the idea was to let the ulp post --two-- work requests, where the 
first creates the mapping and the second sends this mapping to the 
remote side, such that the second does not start before the first 
completes (i.e a fence).

Now, the above scheme means that the ulp knows the value of the 
rkey/stag at the time of posting these two work requests (since it has 
to encode it in the second one), so something has to be clarified re the 
rkey/stag here, do they change each time this MR is used? how many bits 
can be changed, etc.

I guess my questions are to some extent RTFM ones, but, first, with some 
quick looking in the IB spec I did not manage to get enough answers 
(pointers appreciated...) and second, you are proposing an 
implementation here, so I think it makes sense to review the actual 
usage model to see all aspects needed for ULPs are covered...

Talking on usage, do you plan to patch the mainline nfs-rdma code to use 
these verbs?

Or.
> - MR deallocated with ib_dereg_mr()
> - page lists dealloced via ib_free_fast_reg_page_list()