[openib-general] Re: CMA stuff

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> Michael S. Tsirkin wrote:
>>> The QP is in the INIT state after rdma_create_qp() is called.
>>> Receives may be posted at that time.
>> 
>> I'm talking about the passive side.
> 


The CMA model assumes that the active side initializes the QP
based upon what it intends to request and that the QP is allocated
*before* the connection request is sent to the passive side.

On the passive side the application listens, and receives
connection requests *before* it creates/selects/configures
the QP that it accepts the connection request with.

If anything in the passive side's response private data
requires the QP to be modified then the active side must
do so *after* the connection is established.

That model was the basis for the first versions of both
IT-API and DAT. It is simple for application developers
to understand and maps very well to both transports.

There isn't a lot more that can be done in a transport
neutral way without really complicating things. And
transport dependent connection setup options are NOT
being blocked or even deprecated.

The CMA model is limited, but it provides a safe harbor
transport neutral solution that will work for almost
all applications.

If you have other features that you want, then propose
them and we can kick the tires to make sure that they
map to all of the relevant devices. But be warned that
the RTR state was not well documented in the RDMAC
verbs. This has a major impact on when the passive
side can report "connection established" through
the callback and thereby enabling posts to the Send
Queue. 

 
>> My idea was to have 2 counters: one counting connections that are
>> being established, gong up to tcp_max_syn_backlog, and another one
>> after the connection is established, up to local socket backlog.
> 
> My understanding is that with sockets, backlog is the number
> of queued connections not yet accepted by the user.  With the
> CMA, connection requests are explicitly accepted by the user
> before the connection is established.

It is before the RDMA connection is established, but it would
be *after* a TCP connection is established. The MPA Request
Frame arrives over the TCP connection.

> Connection requests are reported to the user directly through
> a callback, rather than being queued until the user comes to claim
> them. 

Correct. Which means that you *could* dispense with the
'backlog' parameter entirely and simply have a "busy"
response on the callback. When the consumer cannot accpet
a callback the connection request should be rejected by
the CMA itself (for iWARP there is a specific non-peer
rejection message for this).

That works. A backlog parameter was proposed because
it also works and is more in line with what sockets
developers are already familiar with.