[openib-general] mthca FMR correctness (and memory windows)

[Caitlin Bestler]

ftillier.sst at gmail.com wrote:
> On 3/21/06, Caitlin Bestler <caitlinb at broadcom.com> wrote:
>> openib-general-bounces at openib.org wrote:
>>> 
> ...snip...
>>> 
>>> I just want the fastest possible map and unmap. I guess that means
>>> I want fast MTT's. 
>>> 
>> 
>> Which is exactly what an FMR work request wouuld provide, which is
>> the interface specified by both RDMAC and IB 1.2.
>> 
>> When an FMR is a work request it *must* be lightweight (no resource
>> allocation, just copying new pre-validated already prepped values
>> into existing resources) and can be fully pipelined.
> 
> Memory window bind operations are work requests, but they're
> not light weight - they stall the send queue until the bind
> is complete, as Tom pointed out.  Do FMRs behave in a similar way?
> 
> - Fab

Actually, compared to work requests that generate output they
complete annoyingly fast. A window bind involves less checking
than a send request does, and the only real work required is
to update what slice of existing PBLEs the window now references.
So the only sense that they "stall" is that you have to fetch
them, and if there are more binds in a row than you pre-fetch
you could have a slight "stall". 

So the only real impact of MWs and FMRs on QP performance is
that there are more work requests. But the alternative, using
larger SGLs, probably results in less efficient use of the
send queue ring.

We shouldn't design the API around limitations of any
specific implementation. There is NO inherent reason for
MW binds, or FMR work requests, to be inefficient.