[openib-general] mthca FMR correctness (and memory windows)

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
>     Thomas> Yes, I know about binding on a separate queue. That
>     Thomas> doesn't work, because windows are semantically not
>     Thomas> fungible (for security reasons).
> 
> Can you elaborate on the issue of fungibility?  If one entity
> has two QPs, one of which it's using for traffic and one of
> which it's using for MW binds, I don't see any security issue
> (beyond the fact that you've now given up ordering of operations
> between the QPs). 
> 

A narrow memory window makes it easy for the provider to guarantee
that all use of an STag has ceased when it reports a remote invalidation
of that STag.

If an STag can be in use on multiple QP then reporting the completion
must be delayed until any activity using the STag on another QP can
complete. Checking state across QPs is a very expensive operation.

RNIC-PI formalized that this guarantee was not applicable to wide
memory windows, and it is not relevant to the usage models that
benefit from wide memory windows. Adopting that model would work
(unless we're afraid that brain-deadness is contagious).

A closely releated refinement comes from the IETF's security 
draft that states that shared STags should not be remotely
invalidatable.