[ewg] Re: [ofa-general] Re: [PATCH] Request For Comments:

[Caitlin Bestler]

On Tue, May 6, 2008 at 11:32 AM, Steve Wise <swise at opengridcomputing.com> wrote:
> Roland Dreier wrote:
>
> >  > - always do peer2peer and don't let the app choose.  This forces
> >  > the overhead of p2p mode on all apps, but preserves the API.
> >
> > How bad is the overhead?
> >
> >  - R.
> >
> >
>  The client side must send a "Ready To Receive" message.  This will be
> negotiated via the MPA exchange and the resulting RTR message may be a 0B
> read + read response, 0B write, or a 0B send.  For chelsio, the 0B write
> couldn't be used, and the 0B read was the least impact on the driver code,
> so we used that.  For nes, they currently use a 0B write.
>
>  Also, there are some "caveats" if you turn this on:
>
>  1) private data is used to negotiate the type of RTR message and if its
> needed.   This is more of a global module option I think, since it will
> break interoperability with iwarp.  Prolly will bump the MPA version number
> if this option is on too.
>
>  2) if the RTR message fails, it can generate a CQE that is unexpected.
>
>  3) if using SEND, then a recv completion is always generated.
>
>  Steve.
>
>
>

Keep in mind that even if it is a zero byte RDMA Write, it is still a distinct
packet that needs TCP handling, will occupy a buffer in various switch
queues, etc.

So while it can be about as innocuous as any TCP segment can be, it
is still an excess packet if it did not need to be sent. The overwhelming
majority of applications use a client/server model rather than peer2peer.
For them this is an excess wire packet, so I think that would make it
excessive overhead.

Secondly, the applications that need this feature will generally know
that they need it. Developers of MPI and other peer-2-peer applications
tend to know advanced networking a bit more than typical app developers.
So keeping the default to match the client/server model makes sense.