[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Steve Wise]

Or Gerlitz wrote:
> Steve Wise wrote:
>> So you allocate the rkey/stag up front, allocate page_lists up front, 
>> then as needed you populate your page list and bind it to the 
>> rkey/stag via IB_WR_FAST_REG_MR, and invalidate that mapping via 
>> IB_WR_INVALIDATE_MR.  You can do this any number of times, and with 
>> proper fencing, you can pipeline these mappings.   Eventually when 
>> you're done doing IO (like for NFSRDMA when the mount is unmounted) 
>> you free up the page list(s) and mr/rkey/stag.
> Yes, that was my thought as well.
> 
> Just to make sure, by "proper fencing" your understanding is that for 
> both IB and iWARP the ULP should not wait for the fmr work request to 
> complete and post the send work-request carrying the rkey/stag with the 
> IB_SEND_FENCE flag?
> 
> Looking in the IB spec, its seems that the fence indicator only applies 
> to previous rdma-read / atomic operations, eg in section  11.4.1.1 POST 
> SEND REQUEST it says:
>> Fence indicator. If the fence indicator is set, then all prior RDMA 
>> Read and Atomic Work Requests on the queue must be completed before 
>> starting to process this Work Request.
> 

The fast register and invalidate work requests require that they be 
completed by the device _before_ processing any subsequent work 
requests.  So you can post subsequent SEND WRs that utilize the rkey 
without problems.

In addition, invalidate allows a local fence which means the device will 
no begin processing the invalitdae until all _prior_ work requests 
complete (similar to a read fence but for all prior WRS).

Steve.