[openib-general] mthca FMR correctness (and memory windows)

[Talpey, Thomas]

At 06:00 PM 3/20/2006, Sean Hefty wrote:
>Can you provide more details on this statement?  When are you fencing the send 
>queue when using memory windows?

Infiniband 101, and VI before it. Memory windows fence later operations
on the send queue until the bind completes. It's a misguided attempt to
make upper layers' job "easier" because they can post a bind and then
immediately post a send carrying the rkey. In reality, it introduces bubbles
in the send pipeline and reduces op rates dramatically.

I argued against them in iWARP verbs, and lost. If Linux could introduce
a way to make the fencing behavior optional, I would lead the parade.
I fear most hardware is implemented otherwise.

Yes, I know about binding on a separate queue. That doesn't work,
because windows are semantically not fungible (for security reasons).

Tom.