[ofa-general] post_recv question

[Caitlin Bestler]

On Thu, Feb 21, 2008 at 10:36 AM, Jeff Squyres <jsquyres at cisco.com> wrote:
> On Feb 21, 2008, at 10:19 AM, Gleb Natapov wrote:
>
>  >> To further complicate things, this race condition is never seen
>  >> _if_ the
>  >> application uses the same QP to advertise (send a credit allowing the
>  >> peer to SEND) the RECV buffer availability.  So if the app posts a
>  >> SEND
>  >> after the RECV is posted and that SEND allows the peer access to the
>  >> RECV buffer, then everything is ok.  This is due to the fact that the
>  >> FW/HW will process the SEND only after processing the RECV.  If the
>  >> app
>  >> uses a different QP to post the SEND advertising the RECV, then the
>  >> race
>  >> condition exists allowing the peer to SEND into that RECV buffer
>  >> before
>  >> the HW makes it ready.
>  >>
>  > OpenMPI can be configured to send credit updates over different QP.
>  > I'll
>  > try to stress it next week to see what happens.
>
>
>  FWIW: this is exactly where the question arose: Steve's working on the
>  iwarp port of OMPI, and since we send the flow control messages for
>  all QP's between a pair of processes over a single QP, this apparent
>  race condition can occur.
>

I believe that the iWARP RFCs are clear that SEND/RECV flow control
is the responsibility of the ULP.

Because it is the ULP's responsibility, the ULP may use ANY method
of communicating that flow control. It does not have to be the same connection,
it does not even have to be the same network.

Aside from the oft-cited carrier pigeon method of delivering credits, there
are ULPs that use multiple connections for reliability. In those a credit
would typically be implicitly granted to send a reply with each request,
but the reply could occur on a different connection within the session.