[openib-general] Immediate data question

[Michael Krause]

At 09:10 PM 2/11/2007, Devesh Sharma wrote:
>On 2/10/07, Tang, Changqing <changquing.tang at hp.com> wrote:
>> > >
>> > >Not for the receiver, but the sender will be severely slowed down by
>> > >having to wait for the RNR timeouts.
>> >
>> > RNR = Receiver Not Ready so by definition, the data flow
>> > isn't going to
>> > progress until the receiver is ready to receive data.   If a
>> > receive QP
>> > enters RNR for a RC, then it is likely not progressing as
>> > desired.   RNR
>> > was initially put in place to enable a receiver to create
>> > back pressure to the sender without causing a fatal error
>> > condition.  It should rarely be entered and therefore should
>> > have negligible impact on overall performance however when a
>> > RNR occurs, no forward progress will occur so performance is
>> > essentially zero.
>>
>>Mike:
>>         I still do not quite understand this issue. I have two
>>situations that have RNR triggered.
>>
>>1. process A and process B is connected with QP. A first post a send to
>>B, B does not post receive. Then A and B are doing a long time
>>RDMA_WRITE each other, A and B just check memory for the RDMA_WRITE
>>message. Finally B will post a receive. Does the first pending send in A
>>block all the later RDMA_WRITE ?
>According to IBTA spec HCA will process WR entries in strict order in
>which they are posted so the send will block all WR posted after this
>send, Until-unless HCA has multiple processing elements, I think even
>then processing order will be maintained by HCA
>If not, since RNR is triggered

The source HCA is responsible for processing work requests in the order 
they are posted.   If the SEND cannot proceed and receives a RNR, then the 
subsequent RDMA Write should not proceed, i.e. the sequence numbers that 
define the valid window will not progress and given IB requires strong 
ordering within the fabric, nothing sent subsequently should be made 
visible at the sink HCA.   In your example, if A is sending a SEND followed 
by a RDMA Write, the first check should have been that B had provided an 
ACK with a credit indicating that a SEND is allowed.  If B subsequently 
removed access to the buffer that had to be posted to provide that credit, 
then it should trigger a RNR NAK and the subsequent RDMA Writes should not 
be visible at B since there is no an effective hole in the transmission stream.

>>periodically till B post receive, does it affect the RDMA_WRITE
>>performance between A and B ?
>>
>>2. extend above to three processes, A connect to B, B connect to C, so B
>>has two QPs, but one CQ. A posts a send to B, B does not post receive,
>>rather B and C are doing a long time RDMA_WRITE, or send/recv. But B
>>must sends RNR periodically to A, right?. So does the pending message
>>from A affects B's overall performance  between B and C ?

Neither IB nor iWARP provide any ordering guarantees between different data 
flows.  This is strictly under application control.  Hence, if a RNR NAK or 
whatever occurs on a RC between A and B, then it has no impact on what 
occurs between A and C or B and C.   It is simply outside the scope of 
either technology to address.

Mike