[openib-general] RFC: CMA backlog

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> Hello, Sean!
> I am looking at implementing the listen backlog parameter correctly.
> Here's what this does in TCP: TCP counts the number of
> connect requests at the specific local socket that were not
> yet accepted by accept().
> Once this number exceeds the backlog specified in listen, new
> SYN packets will be dropped, and the remote side will retry.
> 
> Implementing this correctly is important for scalability when
> a lot of clients connect to a single server - this way if
> there is a pressure on server to handle connections, it is
> distributed between clients who have to retry.
> 
> Currently, this is not something that can be implemented by
> ULP on top of CMA, because returning error from REQ will
> result in reject rather than REQ drop.
> 
> CMA already has backlog parameter in listen but it is ignored
> as far as I can see. I propose extending cma API with the
> following options:
> 
> rdma_backlog_added - connection was added to backlog queue
> rdma_backlog_removed - connection was removed from backlog queue
> 
> Internally, CMA will count the # of connections in backlog.
> If If REQ arrives and this number exceeds the backlog given
> in listen, CMA will drop the REQ, without creating the new CMA ID.
> 
> This mimics the TCP behaviour closely an will hopefully scale well.
> 
> Opinions?

Dropping is one option, but sending an explicit non-peer Reject
certainly makes sense for iWARP.

In any event, a Reject from the user is a peer reject. No matter
what the protocol, the CMA has to be able to ignore or reject
requests that exceed the number it can submit to the user for
approval.

The other clarification is that the connectino request counts
against the total until it is accepted *or* rejected.

I'm not following what the options actually do, clearly there
is a need to have a count of pending connection requests per
listen. And once that credit is exhausted there are no more
connection requests generated (either by reflex rejection
or by dropping). Are you suggesting something beyond that?