[ofa-general] Re: [PATCH] Request For Comments:

[Steve Wise]

Sean Hefty wrote:
>>>  > nes - requires 0b write
>>>  > cxgb3 - requires 0b read
>>>  > amso1100 - won't work in p2p mode
> 
> I'm assuming by requires that you, uhm, mean requires, and nes couldn't do 0b
> reads, or cxgb3 0b writes.
> 
Well, I'm not sure about nes.  But cxgb3 cannot deal with receiving a 0B 
write for the RTR because the FW doesn't see incoming writes, nor does 
the driver.

nes may be able to request a 0b read, but I what I meant was they 
currently use a 0B write and not a read.

So its possible to reduce the complexity if we just mandate 0B read for 
RTR.  But it makes sense in my mind to allow the other message types...

>> Its is painful.  But without anything, you cannot run OMPI, IMPI or
>> HPMPI on a iwarp cluster with mixed vendor rnics...
> 
> Is there any requirement at the receiving side, versus the initiating side?
> That is, just because nes issues a 0b write, does the receiving HW care if a
> read or write shows up?  Or is this restriction on both sides?
> 
The requirement is mostly driven from the receiving side.  For cxgb3 it 
is anyway...

The receiving side, ie the side that issues the rdma_accept will tell 
the sending side what RTR message to send, if any.  So the MPA exchange 
will look like this:

client sends MPA Start request with private data saying "i can send an 
RTR if you want it".

server moves connection into RDMA mode

server sends MPA Start response with "lets do RTR and send me X"  where 
X could be 0B write, 0B read request or 0B send.

client moves connection into RDMA mode

client sends X and then enables SQ processing (or indicate ESTABLISHED)

Once server gets X it can enable SQ processing (or indicate ESTABLISHED)

If X was a 0B read request, server sends 0B read response.



Steve