[openib-general] [PATCH][iWARP] Added provider CM verbs and queryprovider methods

[Caitlin Bestler]

Sean Hefty wrote:

> 
> I need to reacquaint myself with iWarp more, but I don't like 
> the idea of adding CM calls as part of the verbs API, and in 
> particular as part of a generic RDMA device structure.
> 
> Does this suggest that each iWarp device driver will need to 
> implement a connection establishment protocol?  Isn't there a 
> way to generalize that into a single iWarp CM module that can 
> sit above multiple devices?  How will connections between 
> different devices be supported?
> 
> I'm assuming that the Linux kernel will never permit an 
> established connection to be offloaded onto a NIC.  However 
> it seems possible that a new iWarp connection could be done 
> in a common way, with the result passed into the device 
> through the modify QP call as the LLP stream.
> 

It is indeed possible to build a device-independent iWARP
Connection Manager on top of sockets and the 'modify qp 
to rts' verb request.

However, this requires standardizing:

a) how you open a socket that will later be used for RDMA.
b) how to use that socket in pre-RDMA mode.
c) how to transition that socket into RDMA mode.

If the socket is not a normal socket used by the host stack
then it must be a SOCK_STREAM socket associated with the
offload device itself. That means you are now publishing
a complete sockets API for TCP connections to the offload
device and then *not* integrating it with the host stack
-- which seems a strange thing to do.

Structuring pre-RDMA mode to be fully compliant with
all system administrative controls without integreting
it with the host stack itself is possible, but it will
slow down the entire process of making complete open
source stacks available.

Providing a "one stop" API that avoids the need to standardize
the interactions between pre-RDMA and RDMA modes for the same
connection allows drivers to be made available sooner, and
provides more time to discuss what the optimal standardization
of pre-RDMA mode actually is.

Finding that optimal interface will take time, because different
implementations vary on how work is divided between the device,
driver and verbs. There is also a very distinct need to preserve
OS control over connection setup. The simplest method of doing
this, having the host stack set up all connections, does not 
seem to be available for a variety of non-technical reasons.
Building a conscensus on the next best option will take time.
Meanwhile listen/connect/accept/reject methods defined on a 
per-device basis will allow virtually all applications to be
deployed even before the long-term stack co-ordination issues
are fully resolved.