[openib-general] [PATCH] [RFC] dapltest change for iwarp

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> Steve Wise wrote:
>> The Chelsio RNIC has this issue.  If the server sends the first FPDU
>> _before_ the client driver has moved the connection/qp into RDMA
>> mode, then the data is placed as streaming data and the connection
>> must be terminated (dapltest 6 exposes this intermittently). Ammasso
>> doesn't have this issue, but other RNIC's probably will.
>> 
>> One thing I'm experimenting with is to delay the ESTABLISHED event on
>> the server side until the first FPDU is received.  However, we still
>> probably need a way for an application to know whether the client has
>> to send first (or the server as you pointed out).
>> 
>> I believe rnic-pi has an attribute that indicated this behavior...
> 
> Can this be hidden from the user?  If the client side needs
> to send the first message, couldn't the connection protocol
> send a 0-byte message that the server strips out?  The
> ESTABLISHED event would be delayed until this message is received.
>

A zero byte Send/Receive message is not transparent, it consumes
a work request. A zero byte write avoids this, and if you read
only the MPA specification you would believe that it would work.
Unfortunately, the RDMAC verbs assumed there was no harm in requiring
the active side to do the first Send, and clearly encourages that
less accomodating approach in the firmware. So relying on a zero
byte RDMA Write to clear the problem away would not be advisable.

Now any given application can avoid this problem entirely by 
simply having a NOP request that can be sent by the active side
when it doesn't have anything to do at startup. But that's a few
layers too high to influence when the Established event is generated.
 
> Also, wouldn't this issue need to be resolved based on both
> the local and remote device capabilities?  I.e. if the server
> is using an Ammasso RNIC, it needs to know the behavior of
> the remote RNIC, in case it's Chelsio.  Is this information
> carried as part of the connection, or does iWarp force the
> client to initiate the first send?
> 

On an IP network you don't know what model NIC your peer is using.

The real quesetion is how the server knows when it can post to
the Send Queue. Having a Provider Attribute allows this to be
handled at the verbs or middleware layer so that the application
can ignore these variations.