[openib-general] posting send requests in RTR

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
>> From: Sean Hefty [mailto:sean.hefty at intel.com]
>> 
>>> Not necessarily. CMA consumer at this point is a rare beast, but
>>> since the issue is not specific to CMA, let's look at some IB
>>> protocols: IPoIB connected mode can simply drop a packet.  So can
>>> SRP.  SDP (potential CMA consumer!) simply never needs to send any
>>> application data before communication established event. 
>>> 
>>> Possibly some ULPs may need an extra queue - so let them handle the
>>> complexity. 
>> 
>> If the ULP isn't sharing a CQ across multiple QPs, it could
> just wait
> to
>> arm the
>> CQ (followed by a poll) after receiving the connection establish
>> event.  I think the shared CQ case is the more difficult one for the
>> ULP to handle, and I don't know how many ULPs this affects.
>> 
>> If you wanted to allow queuing sends in RTR, could this be done
>> entirely in software under verbs?  The difficulty comes if an error
>> occurs while trying to post the sends once the QP transitions to RTS.
>> 
> 
> Actually since WQEs are in memory, while in RTR the verbs driver could
> build the WQEs, post them to the QP, just not issue the doorbells.  


That's an implementation assumption. While it is true for every HCA
and RNIC I am aware of it is still the type of assumption that a good
interface should not make. The other assumption is that the HCA/RNIC
does NOT notice that a QP has contents until the doorbell is rung.

NOT ringing a doorbell does NOT guarantee that the send queue will NOT
be processed. An implemetation that lacked full confidence in its
doorbells
might have a background scan to look for non-empty send queues as a
safety
mechanism.

I still see a very simple split. If the hardware knows that it has the
QP fenced until RTS then there is no harm in letting send WQEs be
generated
in RTR, otherwise it returns an error. Meanwhile, the verbs consumer is 
simply told not to post the the send queue until the Connection
Established
event. Each device can then decide when it is safe to generate the
Connection
Estabished event.