[openib-general] mthca FMR correctness (and memory windows)

[Fabian Tillier]

On 3/20/06, Sean Hefty <mshefty at ichips.intel.com> wrote:
> Talpey, Thomas wrote:
> > This, by the way, is why the fencing behavior of memory windows is so
> > painful. I would much rather take an interrupt on bind completion than
> > fence the entire send queue. But there isn't a standard way to do that,
> > even in iWARP. Sigh.
>
> Can you provide more details on this statement?  When are you fencing the send
> queue when using memory windows?

When you bind a MW, you get the RKEY back immediately.  It is valid
for you to send that RKEY in a send that immediately follows the bind.
 Thus, the bind must fence the send, so that the send doesn't pass it
while the HCA is processing the bind locally (since the bind doesn't
actually put anything on the wire).

See 11.2.8.11 in the IB Spec 1.2, volume 1:

"The Bind operation has a unique ordering rule: any Work Request
posted to a Send Queue subsequent to a Bind must not begin execution
until the Bind operation completes."

- Fab