[openib-general] mthca FMR correctness (and memory windows)

[Roland Dreier]

    Thomas> Uh, no can do! When the RPC layer indicates to NFS that
    Thomas> the reply has been received, it must ensure that the reply
    Thomas> buffers are synchronized and stable, i.e. the rkey is
    Thomas> invalidated. For the same reason, it's unacceptable to
    Thomas> share rkey's, which appears to be another goal of the fmr
    Thomas> pool.

    Thomas> So, for now I'll definitely stick to the direct fmr
    Thomas> map/unmap in the RPC path.

If you want to invalidate each FMR individually, then there's not much
point in using FMRs at all.  Just register and unregister memory regions.

    Thomas> Hmm - okay I might try that! Looking at the code, it
    Thomas> almost looks as if I carefully build a bind WR, I could
    Thomas> maybe just use ib_post_send() to post it... ?

No chance.  You need to implement MW allocation in mthca.  It's not a
ton of work but it hasn't reached the top of anyone's list yet.

 - R.