[ewg] Re: [ofa-general] Re: [PATCH] Request For Comments:

[Steve Wise]

Caitlin Bestler wrote:
> On Tue, May 6, 2008 at 11:32 AM, Steve Wise <swise at opengridcomputing.com> wrote:
>   
>> Roland Dreier wrote:
>>
>>     
>>>  > - always do peer2peer and don't let the app choose.  This forces
>>>  > the overhead of p2p mode on all apps, but preserves the API.
>>>
>>> How bad is the overhead?
>>>
>>>  - R.
>>>
>>>
>>>       
>>  The client side must send a "Ready To Receive" message.  This will be
>> negotiated via the MPA exchange and the resulting RTR message may be a 0B
>> read + read response, 0B write, or a 0B send.  For chelsio, the 0B write
>> couldn't be used, and the 0B read was the least impact on the driver code,
>> so we used that.  For nes, they currently use a 0B write.
>>
>>  Also, there are some "caveats" if you turn this on:
>>
>>  1) private data is used to negotiate the type of RTR message and if its
>> needed.   This is more of a global module option I think, since it will
>> break interoperability with iwarp.  Prolly will bump the MPA version number
>> if this option is on too.
>>
>>  2) if the RTR message fails, it can generate a CQE that is unexpected.
>>
>>  3) if using SEND, then a recv completion is always generated.
>>
>>  Steve.
>>
>>
>>
>>     
>
> Keep in mind that even if it is a zero byte RDMA Write, it is still a distinct
> packet that needs TCP handling, will occupy a buffer in various switch
> queues, etc.
>
> So while it can be about as innocuous as any TCP segment can be, it
> is still an excess packet if it did not need to be sent. The overwhelming
> majority of applications use a client/server model rather than peer2peer.
> For them this is an excess wire packet, so I think that would make it
> excessive overhead.
>
> Secondly, the applications that need this feature will generally know
> that they need it. Developers of MPI and other peer-2-peer applications
> tend to know advanced networking a bit more than typical app developers.
> So keeping the default to match the client/server model makes sense.
>   

What are the overwhelming majority of user mode rdma applications that 
don't assume a peer2peer model?

Steve.