[openib-general] Re: port_num

[Steve Wise]

On Thu, 2006-03-16 at 10:08 -0800, Caitlin Bestler wrote:
> openib-general-bounces at openib.org wrote:
> >> I'd like to suggest that CMA implement its own rule along the lines
> >> of: if port is 0, bind to a port number unused by CMA in this port
> >> space and on this device.
> > 
> > To clarify: a user is permitted to bind to a specific port,
> > but use a wildcard IP address.  Just because a user binds to
> > a port doesn't mean that they also bind to an RDMA device.
> > This permits a single listen across all RDMA devices.
> > If one of those devices is an iWarp device, then the port
> > number selected by the CMA ends up needing to be a true TCP
> > port number.
> >
> 
> Listening on a port across a mixture of RDMA and non-RDMA
> devices (or even dissimiliar RDMA devices) is quite problematic.
> The CONNECT_REQUEST events returned ARE bound to a specific
> device, and must be accepted/rejected to that device.
> 

Agreed.

> Mixing a host stack listen with a CMA listen is particularly
> complex because the connection request will be delivered to
> the consumer by a different method.
> 
> These are the reasons why the RDMAC originally specified that
> the host stack would create *all* connections, and then some
> would be migrated to RDMA mode. Unfortunately the netdev
> team does not share that vision. Respecting that, we basically
> have to decide in an inbound connection request for a given
> IP Address / Port number received on an Ethernet port of an
> iWARP RNIC will be handled by the iWARP stack or forwarded
> to the host stack. You really have to do one or the other,
> or convince netdev to accept the "modify qp to rts with 
> llp socket handle" strategy.
> 

Agreed.  We either tightly integrate with the native stack TCP or not.
And the current consensus is to not integrate tightly. 

> Given the need to post receive buffers before accepting
> a connection, it is actually not all that feasible to have
> an RDMA application that is ready to accept connections
> from *any* RDMA device. The application has to know what
> the RDMA device are in advance, create protection domains
> on each, and probably create memory regions on each. It may
> even want to pre-allocate QPs. It is certainly not going to
> respond to a connection request from "eth7" by finding out
> what the RDMA device for eth7 is, opening it, creating the
> PD, MR and QP and *then* accepting the connection request.

Um, CMA allows this now.  The passive side gets the connection request
and the cm_id for this request is associated with a specific device.  So
the app can then create the device-specific pd, mrs, create the qp and
post recvs, and then call rdma_accept() to accept the connection.

rping and krping do this.  NFSoRDMA does this.


Steve.