[openib-general] [RFC] [PATCH] SRQ API

[Caitlin Bestler]

On 6/15/05, Fab Tillier <ftillier at silverstorm.com> wrote:
> > From: Roland Dreier [mailto:roland at topspin.com]
> > Sent: Wednesday, June 15, 2005 1:43 PM
> >
> > Here's a first stab at the changes to the API in <ib_verbs.h> required
> > to support shared receive queues (SRQs).  SRQs are described in
> > version 1.2 of the IBA spec; specifically, sections 10.2.9 and 11.2.3
> > describe the semantics and verbs interface.
> 
> I'm a bit confused still about how WR get processed.  If a QP goes to the error
> state, does it pull all WRs off of the SRQ?  I would expect not.
> 
> How does one flush WRs from an SRQ?  It seems that to use an SRQ, the user has
> to keep a parallel list of all WRs posted to track them so they may be properly
> freed when the SRQ is destroyed.  This also means the completion path now must
> be serialized with the requesting path so that accesses to this tracking list
> are properly synchronized.
> 

For iWARP each  WR is allocated to a specific QP when it is needed
(i.e. when part of the message arrives, or potenitally when a later
message arrives).

WRs that have been assigned to a QP are flushed when the QP
goes to an error state or is discconnected.

I haven't studied the IB 1.2 spec on SRQs at horrendous lenght, but
I strongly suspect that they match the iWARP SRQ specs fairly
closely since I believe the same people wrote them.

In which case one of the inherited defects is that there is no reliable
method of draining an SRQ. The closest you can do is to do a loopback
connection and send yourself short messages until you fail because the
SRQ is empty.

>From a theoretical viewpoint this is a glaring hole. From a realistic
viewpoint virtually all applications that use SRQs create them and
then keep them until the application exits.