[openib-general] posting send requests in RTR

[Rimmer, Todd]

> From: Sean Hefty [mailto:sean.hefty at intel.com]
> 
> >Not necessarily. CMA consumer at this point is a rare beast, but
since
> the
> >issue
> >is not specific to CMA, let's look at some IB protocols: IPoIB
connected
> mode
> >can simply drop a packet.  So can SRP.  SDP (potential CMA consumer!)
> simply
> >never needs to send any application data before communication
established
> >event.
> >
> >Possibly some ULPs may need an extra queue - so let them handle the
> complexity.
> 
> If the ULP isn't sharing a CQ across multiple QPs, it could just wait
to
> arm the
> CQ (followed by a poll) after receiving the connection establish
event.  I
> think
> the shared CQ case is the more difficult one for the ULP to handle,
and I
> don't
> know how many ULPs this affects.
> 
> If you wanted to allow queuing sends in RTR, could this be done
entirely
> in
> software under verbs?  The difficulty comes if an error occurs while
> trying to
> post the sends once the QP transitions to RTS.
> 

Actually since WQEs are in memory, while in RTR the verbs driver could
build the WQEs, post them to the QP, just not issue the doorbells.  On
RTR->RTS the verbs driver could issue the doorbells.  Of course if an
RTR->ERR transition occurs, the WQE Flushed behavior would need to be
simulated, but of course the WQEs themselves are available to look at
(and for the flushed event mainly the Work request Id is of interest) so
it shouldn't be too hard.  If this approach were taken the verbs driver
could permit sends to be posted in RTR and delay informing the HCA until
the RTS transition.  This type of approach would completely contain the
issue in the HCA driver rather than having CMA consumers need to worry
about it.

Todd Rimmer