[openib-general] posting send requests in RTR

[Caitlin Bestler]

> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst at mellanox.co.il] 
> Sent: Saturday, July 29, 2006 2:55 PM
> To: Rimmer, Todd
> Cc: Caitlin Bestler; Sean Hefty; Or Gerlitz; Roland Dreier; 
> openib-general at openib.org
> Subject: Re: posting send requests in RTR
> 
> Quoting r. Rimmer, Todd <trimmer at silverstorm.com>:
> > The target receives the SCSI commands (such as Test Unit Ready or
> > Inquiry) and wants to act on them immediately.  However if 
> the command 
> > has passed the RTU or the RTU is lost, the target is still 
> in RTR.  If 
> > the command was very simple, the target may want to answer 
> the query 
> > immediately by posting a send with the response.
> 
> Since the response won't go out until QP is in RTS anyway, 
> why is it important to post the send immediately?  The 
> simplest appproach for you is to avoid polling the CQ until 
> QP is in RTS.
> 

That only works if you can avoid polling the CQ until
*the* QP associated with it is in RTS.

But if the CQ supports a large number of QPs then the
application should not be expected to figure out which
receive completions it cannot process because they are
from connections that are not yet established.

Especially since the connections are established, because
otherwise you could not have a successful completion
of a receive work request, but not all of the stack
understands that yet.

The provider should deal with its own confustion and
not inflict it on the consumer. It is very reasonable
to expect the active side to wait for a "connection
established" event before it sends its first message.
It is something totally different to tell a server
application supporting a very large number of connections
that it cannot respond to a request received on one of
those connections because parts of the driver do not
realize that the connection is established yet (although
it really is).

Whether the need for fencing originates in the protocol
(the IETF MPA requirement that the passive side receive
the first MPA frame before it sends one) or in the driver
(the QP state in memory is not promptly updated) the
solution should stay within the Provider and not be
foisted on the application. Getting application developers
to understand RDMA is challenging enough without adding
more rules on top of the ones that are truly needed.