[openib-general] mthca FMR correctness (and memory windows)

[Sean Hefty]

Fabian Tillier wrote:
> When you bind a MW, you get the RKEY back immediately.  It is valid
> for you to send that RKEY in a send that immediately follows the bind.
>  Thus, the bind must fence the send, so that the send doesn't pass it
> while the HCA is processing the bind locally (since the bind doesn't
> actually put anything on the wire).
> 
> See 11.2.8.11 in the IB Spec 1.2, volume 1:
> 
> "The Bind operation has a unique ordering rule: any Work Request
> posted to a Send Queue subsequent to a Bind must not begin execution
> until the Bind operation completes."

I thought that he was referring to fencing required on the part of the user, 
rather than work request ordering requirements.

Why would a bind necessarily be slower than processing any other work request? 
Is this really an architectural issue, or an implementation one?

Also, if you don't want to slow down the entire send queue because of a bind 
operation, couldn't you create a separate QP on the same PD that you use only 
for bind operations?

- Sean