[openib-general] mthca FMR correctness (and memory windows)

[Fabian Tillier]

On 3/20/06, Talpey, Thomas <Thomas.Talpey at netapp.com> wrote:
> At 06:00 PM 3/20/2006, Sean Hefty wrote:
> >Can you provide more details on this statement?  When are you fencing the send
> >queue when using memory windows?
>
> Infiniband 101, and VI before it. Memory windows fence later operations
> on the send queue until the bind completes. It's a misguided attempt to
> make upper layers' job "easier" because they can post a bind and then
> immediately post a send carrying the rkey. In reality, it introduces bubbles
> in the send pipeline and reduces op rates dramatically.
>
> I argued against them in iWARP verbs, and lost. If Linux could introduce
> a way to make the fencing behavior optional, I would lead the parade.
> I fear most hardware is implemented otherwise.

Right - I don't think this has anything to do with the OS, Linux or
otherwise.  If the hardware serializes things internally, it doesn't
matter what the OS wants.  I suspect changing existing hardware
implementations and the specs they implement is exceedingly unlikely.

> Yes, I know about binding on a separate queue. That doesn't work,
> because windows are semantically not fungible (for security reasons).

Even if the two QPs are on the same protection domain?  I thought as
long as you're on the same PD, you can share MRs and MWs.  You as the
app are then responsible for proper synchronization to make sure you
don't advertise the RKEY prematurely.

>From the IB spec:
"The QP, Memory Window, and Memory Region must belong to the same
HCA and Protection Domain."

I didn't see anything that prevented the resulting memory window from
being used on a different QP within the same PD and HCA.  Does iWarp
behave differently?

- Fab