[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Or Gerlitz]

Steve Wise wrote:
> So you allocate the rkey/stag up front, allocate page_lists up front, 
> then as needed you populate your page list and bind it to the 
> rkey/stag via IB_WR_FAST_REG_MR, and invalidate that mapping via 
> IB_WR_INVALIDATE_MR.  You can do this any number of times, and with 
> proper fencing, you can pipeline these mappings.   Eventually when 
> you're done doing IO (like for NFSRDMA when the mount is unmounted) 
> you free up the page list(s) and mr/rkey/stag.
Yes, that was my thought as well.

Just to make sure, by "proper fencing" your understanding is that for 
both IB and iWARP the ULP should not wait for the fmr work request to 
complete and post the send work-request carrying the rkey/stag with the 
IB_SEND_FENCE flag?

Looking in the IB spec, its seems that the fence indicator only applies 
to previous rdma-read / atomic operations, eg in section  11.4.1.1 POST 
SEND REQUEST it says:
> Fence indicator. If the fence indicator is set, then all prior RDMA 
> Read and Atomic Work Requests on the queue must be completed before 
> starting to process this Work Request.

>> Talking on usage, do you plan to patch the mainline nfs-rdma code to 
>> use these verbs?
> Yes.  Tom Tucker will be doing this.  Jon Mason is implementing RDS 
> changes to utilize this too.  The hope is all this makes 2.6.27/ofed-1.4.
>
> I can also post test code (krping module) if anyone is interested.  
> I'm developing that now.
>
Posting this code would be very much helpful (also to the discussion, I 
think), thanks.

Or.