[openib-general] posting send requests in RTR

[Caitlin Bestler]

Rimmer, Todd wrote:
>> From: Caitlin Bestler [mailto:caitlinb at broadcom.com]
>> 
>>> 
>>> Actually since WQEs are in memory, while in RTR the verbs driver
>>> could build the WQEs, post them to the QP, just not issue the
>>> doorbells. 
>> 
>> 
>> That's an implementation assumption. While it is true for every HCA
>> and RNIC I am aware of it is still the type of assumption that a good
>> interface should not make. The other assumption is that the HCA/RNIC
>> does NOT notice that a QP has contents until the doorbell is rung.
>> 
>> NOT ringing a doorbell does NOT guarantee that the send queue will
>> NOT be processed. An implemetation that lacked full confidence in its
>> doorbells might have a background scan to look for non-empty send
>> queues as a safety mechanism.
> 
> The assumption would not be in the interface, but rather in
> the hardware specific implementation of the verbs driver.
> Hence the appropriate assumption could be made for each
> hardware model.  As needed the alternate TX queue could be
> implemented in the verbs driver.
> 
>> 
>> I still see a very simple split. If the hardware knows that it has
>> the QP fenced until RTS then there is no harm in letting send WQEs be
>> generated in RTR, otherwise it returns an error. Meanwhile, the verbs
>> consumer
> is
>> simply told not to post the the send queue until the Connection
>> Established event.
> 
> The challenge is that every consume must then implement a
> alternate TX queue to handle this situation.  It would be
> much better to hide this race in the stack or the verbs
> driver rather than expose it for error prone implementation
> in every consumer (both user space and kernel consumers would be
> vulnerable). 
> 
> Todd Rimmer

That assumes that there is any valid reason for an application to
post send requests before the connection is established. While there
is clearly a need to post receive work requests before the connection
is established I cannot think of any reason why an application needs
to pre-prime the send queue.

Putting unneeded complexity in the definition of a hardware service
just invites more hardware dependencies and eventual hardware specific
bugs that will complicate life for application developers. "Don't post
until the connection is established" is very simple for *both* the
application and the verbs provider.