[openib-general] posting send requests in RTR

[Rimmer, Todd]

> From: Caitlin Bestler [mailto:caitlinb at broadcom.com]
> 
> >
> > Actually since WQEs are in memory, while in RTR the verbs driver
could
> > build the WQEs, post them to the QP, just not issue the doorbells.
> 
> 
> That's an implementation assumption. While it is true for every HCA
> and RNIC I am aware of it is still the type of assumption that a good
> interface should not make. The other assumption is that the HCA/RNIC
> does NOT notice that a QP has contents until the doorbell is rung.
> 
> NOT ringing a doorbell does NOT guarantee that the send queue will NOT
> be processed. An implemetation that lacked full confidence in its
> doorbells
> might have a background scan to look for non-empty send queues as a
> safety
> mechanism.

The assumption would not be in the interface, but rather in the hardware
specific implementation of the verbs driver.  Hence the appropriate
assumption could be made for each hardware model.  As needed the
alternate TX queue could be implemented in the verbs driver.

> 
> I still see a very simple split. If the hardware knows that it has the
> QP fenced until RTS then there is no harm in letting send WQEs be
> generated
> in RTR, otherwise it returns an error. Meanwhile, the verbs consumer
is
> simply told not to post the the send queue until the Connection
> Established
> event.

The challenge is that every consume must then implement a alternate TX
queue to handle this situation.  It would be much better to hide this
race in the stack or the verbs driver rather than expose it for error
prone implementation in every consumer (both user space and kernel
consumers would be vulnerable).

Todd Rimmer