[openib-general] mthca FMR correctness (and memory windows)

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> At 06:00 PM 3/20/2006, Sean Hefty wrote:
>> Can you provide more details on this statement?  When are you fencing
>> the send queue when using memory windows?
> 
> Infiniband 101, and VI before it. Memory windows fence later
> operations on the send queue until the bind completes. It's a
> misguided attempt to make upper layers' job "easier" because
> they can post a bind and then immediately post a send
> carrying the rkey. In reality, it introduces bubbles in the
> send pipeline and reduces op rates dramatically.
> 
> I argued against them in iWARP verbs, and lost. If Linux
> could introduce a way to make the fencing behavior optional, I would
> lead the parade. I fear most hardware is implemented otherwise.
> 
> Yes, I know about binding on a separate queue. That doesn't
> work, because windows are semantically not fungible (for security
> reasons). 
> 
> Tom.
> 

If Memory Windows are efficient enough they don't stall
the pipeline. In any event it is far more of a stall to
have to set up MRs, MWs and FMRs outside the scope of
the pipeline.

Having Linux specify hardware is not a good idea. It has
enough on its hands understanding the hardware that is
already developed. Having Linux developers take part in
industry forums where things like Verbs are developed
on an industry-wide basis would be an excellent idea
in future rounds.

The rationale for narrow memory windows is that it makes
per QP caching of state far more efficient, which makes
providing security easier. Providing deterministic 
cessation of the use of an invalidated RKey/STag is
much easier when the scope of potential use of the
RKey/STag is confined to the single QP. Since this
corresponds to 90% or more of the usage, optimizing
for this case makes sense. The major oversight in 
the RDMAC verbs (and IB 1.2) is that all of this
rationale for narrow memory windows is fully applicable
(or more so) to FMRs, which are unfortunately defined
as wide (although the terms aren't used, the narrow/
wide terminology came after the fact from IT-API).