[openib-general] mthca FMR correctness (and memory windows)

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> Ok, this is a longer answer.
> 
> At 06:08 PM 3/20/2006, Fabian Tillier wrote:
>> You pre-alloc the MPT entry, but not the MTT entries.  You then
>> populate the MTT by doing posted writes to the HCA memory (or host
>> memory for memfree HCAs). ...
>> I don't know if allocating MTT entries is really expensive. What
>> costs is the fact that you need to do command interface transactions
>> to write the MTT entries, while FMRs support posted writes.
> 
> I don't know what MPTs and MTTs are (Mellanox
> implementation?) nor do I know exactly what the overhead
> difference you refer to really is.
> It's less about the overhead and more about the resource contention,
> in my experience. 
> 
>> That is, just like with alloc_fmr, you need to reserve and format an
>> MPT for regular memory registrations, which is a command interface
>> transaction.  For memory registration, one or more commands precede
>> this to write to the MTT. Thus, a memory registration is at a
>> minimum a 2 command interface transaction operation, potentially
>> more depending on the size of the registration.
>> 
>> Deregistration and freeing (not unmapping) an FMR should be
>> equivalent, I would think.
> 
> So, in the RPC/RDMA client, I do ib_alloc_fmr() a bunch of
> times way up front, when setting up the connection. This
> provides the "windows" which are then used to register chunks
> (RPC/RDMA segments). 
> 
> As each RPC is placed on the wire, I borrow fmr's from the
> above list and call ib_map_phys_fmr() to establish the
> mapping for each of its segments.
> No allocation is performed on this hot path.
> 
> When the server replies, I call ib_unmap_fmr() to tear down
> the mappings.
> No deallocation is performed, the fmrs are returned to a
> per-mount pool, *after unmaping them*.
> 
> I just want the fastest possible map and unmap. I guess that means I
> want fast MTT's. 
> 

Which is exactly what an FMR work request wouuld provide,
which is the interface specified by both RDMAC and IB 1.2.

When an FMR is a work request it *must* be lightweight (no resource
allocation, just copying new pre-validated already prepped values
into existing resources) and can be fully pipelined.

Work requests are the only vendor neutral, transport neutral,
solution that is viable here.