[openib-general] Re: [PATCH] [RFC] - example user moderdmaping/pongprogram using CMA

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> At 11:35 AM 2/8/2006, Steve Wise wrote:
> 
> 
> 
> 	> >
> 	> > I just read this section in the 1.2 version of the
> spec, and I still
> 	> > don't understand what the issue really is?  9.7.7.2
> talks about IBA
> 	> > doing flow control based on the RECV WQEs posted.
> rping always ensures
> 	> > that there is a RECV posted before the peer can
> send.  This is ensured
> 	> > by the rping protocol itself (see the comment at
> the front of rping.c
> 	> > describing the ping loop).
> 	> >
> 	> > I'm only ever sending one outstanding message via SEND/RECV.
I
> 	would > > rather post exactly what is needed, than post some
> number of RECVs "just
> 	> > to be safe".  Sorry if I'm being dense.  What am I
> missing here?
> 	> >
> 	> > Steve.
> 	> >
> 	>
> 	> As far as I know, the credits are only updated by the
> ACK messages.
> 	> If there is a single work request outstanding on the RQ,
> 	> the ACK of the SEND message will have the credit field value 0
> 	> (since exactly one receive WR was outstanding, and
> that is now consumed).
> 	>
> 	> As a result the remote side withh "think" that there are no
> 	> receive WQEs and will slow down (what spec refers to
> as limited WQE).
> 
> 	Oh.  I understand now.  This is an issue with only 1 RQ
> WQE posted and
> 	how IB tries to inform the peer transport of the WQE
> count.  For iWARP,
> 	none of this transport-level flow control happens (and
> I'm more familiar
> 	with iWARP than IB).
> 
> 
> For iWARP, we decided to not implement application receiver
> based flow control due to two items:TCP provides
> transport-level flow control (IB does not provide the
> equivalent per se) and upon examination of the majority of
> the ULP, they exchange and track the number of receive
> buffers allowed to be processed thus there is no need to
> replicate this in iWARP.  There are some subtleties as well
> between a message-based transport and a byte stream such as
> TCP that go into the equation but these are not that
> important for most application writers to deal with.
> 
> Mike


But in terms of compiling the safe harbor transport neutral
recommended programming practices, I think this is a valid
point. Having one "spare" buffer is a good safety mechanism
at the application layer in general, *and* it may prevent
snarls in the transport layer flow control. 

Suggesting that consumers avoid letting the RQ hit empty
strikes me as aa valid transport neutral recommendation.
And we'll improve the public education by following those
recommendations in sample and test programs.