[openib-general] mthca FMR correctness (and memory windows)

[Michael Krause]

At 04:30 PM 3/20/2006, Talpey, Thomas wrote:
>At 06:00 PM 3/20/2006, Sean Hefty wrote:
> >Can you provide more details on this statement?  When are you fencing 
> the send
> >queue when using memory windows?
>
>Infiniband 101, and VI before it. Memory windows fence later operations
>on the send queue until the bind completes. It's a misguided attempt to
>make upper layers' job "easier" because they can post a bind and then
>immediately post a send carrying the rkey. In reality, it introduces bubbles
>in the send pipeline and reduces op rates dramatically.

The requirement / semantics were derived from the ULP being used to 
construct the technology.  The combination of a bind-n-send operation was 
to reduce the software interactions with the device by consolidating this 
into a combo operation.  I do not follow your logic that this creates a 
bubble in the send pipeline as there were also ordering and correctness 
issues w.r.t. subsequent operations to the send.  The bind-n-send is a 
single operation and its fence semantics were required to allow the bind to 
complete before informing the remote of the subsequent information in order 
to avoid race conditions.

>I argued against them in iWARP verbs, and lost. If Linux could introduce
>a way to make the fencing behavior optional, I would lead the parade.
>I fear most hardware is implemented otherwise.

Hardware generally implements operations in the order they are posted to a 
given QP, i.e. it is a serial execution flow that allows pipelined 
operations to be posted and executed by the hardware.  Scaling is achieved 
by executing across a set of QP and thus a set of resources.  The ordering 
domain requirements are kept simple to allow low-cost hardware 
implementations.  This does not preclude software from executing across a 
set of QP in any order that it desires.

>Yes, I know about binding on a separate queue. That doesn't work, because 
>windows are semantically not fungible (for security reasons).

You could always simply allow a region to be accessible across multiple 
operations but then again storage argued that it must only be accessible 
for a single op thus things like FMR, bind-n-send, etc. were all 
created.  To say that storage was not listened to or their needs were not 
met or balanced against what is practical to implement in either the 
creation of IB or iWARP is simply incorrect.

Mike


>Tom.
>
>_______________________________________________
>openib-general mailing list
>openib-general at openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit 
>http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060323/95ba2bce/attachment.html>