[openib-general] Re: RFC: e2e credits

[Michael Krause]

At 03:31 AM 3/24/2006, Hal Rosenstock wrote:
>On Thu, 2006-03-23 at 12:34, Michael S. Tsirkin wrote:
> > Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > > > > > > > > > >Sean, just to wrap it up, the API at the verbs layer 
> will look
> > > > > > > > > > >like the below, and then ULPs just put the value they 
> want in
> > > > > > > > > > >the CM and CM will pass it in to low level.
> >
> > So this is our question, right?
> >
> >
> > CM REQ and REP messages include the following field:
> >
> > ---------------
> > 12.7.26 END-TO-END FLOW CONTROL
> > Signifies whether the local CA actually implements End-to-End Flow Control
> > (1), or instead always advertises .invalid credits.(0). See section
> > 9.7.7.2 End-to-End (Message Level) Flow Control for more detail.
> > ---------------
> >
> > Consider and implementation that advertises valid credits for
> > connections, and always advertises invalid credits for other connections.
> > This is compliant since the IB spec says (end-to-end (message level) flow
> > control, Requester Behaviour):
> > "Even a responder which does generate end-to-end credits may choose to 
> send the
> > 'invalid' code in the AETH"
>
>I did some spec reading to find this and found the following which I
>think makes the current requirement clear:
>
>p.347 line 37 states "HCA receive queues must generate end-to-end
>credits (except for QPs associated with a SRQ), but TCA receive queues
>are not required to do so. This appears to be informative text. I first
>found the following:
>
>p. 348 has the compliances for this for both HCAs and TCAs:
>
>C9-150.2.1: For QPs that are not associated with an SRQ, each HCA re-
>ceive queue shall generate end-to-end flow control credits. If a QP is
>associated with an SRQ, the HCA receive queue shall not generate end-to-
>end flow control credits.
>
>o9-95.2.1: Each TCA receive queue may generate end-to-end credits ex-
>cept for QPs that are associated with an SRQ. If a TCA supports SRQ, the
>TCA must not generate End-to-End Flow Control Credits for QPs associ-
>ated with an SRQ.
>
>C9-151: If a TCA's given receive queue generates End-to-End credits,
>then the corresponding send queue shall receive and respond to those
>credits. This is a requirement on each send queue of a CA.
>
>The above informative text also references the CA requirements in
>chapter 17 and on p. 1026 line 25 there is a row in the table for end to
>end flow control for RC consistent with the above. p.1028 has the
>compliances for this.
>
> > Is it compliant for CM implementations to set/clear the End-to-End Flow 
> Control
> > field accordingly, taking it to mean
> >
> > "whether the local CA actually implements End-to-End Flow Control
> > (1), or instead always advertises 'invalid credits'(0)"
> > *for the specific connection*
>
>So IMO the intent of what was written is clear (on a per CA basis) and
>this is a spec change which is OK to propose but needs a different
>writeup.

The spec was written to minimize the impact to TCA in terms of e2e 
credits.  HCA are expected to use e2e credits all the time sans SRQ which 
is a special multiplexing case where e2e isn't that beneficial / logical to 
support.  Each connection must negotiate this per CA pair as no single 
policy was deemed practical across all usage models.   I don't think there 
would be much support for changing the spec.

BTW, iWARP does not support e2e credits as it relies upon the ULP to 
advertise and track its buffer usage.  It was therefore deemed unnecessary.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060324/7f0c7305/attachment.html>