[openib-general] RE: cma header - change some things according to the list feedback

[Caitlin Bestler]

Comments imbedded. 

> -----Original Message-----
> From: Guy German [mailto:guyg at voltaire.com] 
> Sent: Thursday, August 25, 2005 7:07 AM
> To: openib-general at openib.org
> Cc: sean.hefty at intel.com
> Subject: cma header - change some things according to the 
> list feedback
> 

> 
> /**
>  * ib_cma_get_device - Returns the device to be used according to
>  *   the destination ip address (this can be detemined according 
>  *   to the local routing table). Call this function before 
>  *   creating the qp. If using link-local IPv6 addresses
>  *   there is no need to call this function.
>  * @remote_address: The destination address for connection
>  * @device: The device to use (returned by the function)  */ 
> int ib_cma_get_device(struct sockaddr *remote_address,
> 		      enum ib_qos qos, struct ib_device **device);
> 
>

Who implements this function? It has to be a core (device-independent)
function, doesn't it? If so what data is it based upon?

I believe the ultimate answer here is that each IP Interface
has to be mapped to 0..1 RDMA devices (or put the other way,
each RDMA device claims exclusive "RDMA enhancement" rights
for one or more IP interfaces).

Once that data is linked the above function can be implemented
by core software or the user. Since it is of general utility
it definitely makes sense to provide the common function.
 
> /**
>  * ib_cma_connect - this is the connect request function, called by 
>  *   the active side. The consumer registers an upcall that will be 
>  *   initiated by the cma with an appropriate connection event 
>  *   notification (established/rejected/disconnected etc)
>  * @cma_conn: This structure contains the following 
> connection parameters:
>  *   @qp: qp for establishing the connection
>  *   @qp_attr: only relevant attributes are used
>  *   @dst_ip: destination ip address
>  *   @service_id: destination service id (port)
>  *   @recv_wr: An array of work requests to post on the 
> receive queue.before 
>  *     sending the RTU
>  *   @num_recv_wr: recv_wr array length - number of buffers 
> to post recv
>  *   @context: context to be returned in the callback
>  *   @cma_event_handler: the upcall function for the active side
>  *   @private_data: private data to be received at the listener upcall
>  *   @private_data_len: private data length (max 255)
>  *   @timeout: 
>  *   @qos: Quality os service for the rc
>  *   @connect_flags: default or multipath connection
>  * @cma_id: This returned handle is a union (different in ib 
> and iwarp)
>  *   in ib - it is the cm_id.
>  */
> int ib_cma_connect(struct ib_cma_conn *cma_conn,
> 		   union ib_cma_id *cma_id);
> 
>

What event is generated at the timeout? Should it be expicitly
guaranteed that each connect call will ultimate produce exactly
one callback?
 
> /**
>  * ib_cma_disconnect - this function disconnects the rc. It can be 
>  *   called, by either the passive or active side
>  * @qp: the connected qp to disconnect
>  * @cma_id: On the active side- this handle is the one returned 
>  *   when ib_cma_connect was called.
>  *   On the passive side- this handle was accepted in 
> cma_listen callback
>  */
> int ib_cma_disconnect(struct ib_qp *qp, union ib_cma_id *cma_id);
> 
> 
> /**
>  * ib_cma_sid_listen - this function is called by the passive 
> side. It is
>  *   listening on a the specified port (ib service id) for incomming 
>  *   connection requests
>  * @device:
>  * @address:
>  * @service_id: service id (port) to listen on
>  * @context: user context to be returned in the callback
>  * @cm_listen_handler: the listen callback
>  * @cma_id: cma handle for the passive side  */ int 
> ib_cma_sid_listen(struct ib_device *device, struct sockaddr *address,
> 		      __be64 service_id, void *context,
> 		      ib_cma_listen_handler cm_listen_handler,
> 		      union ib_cma_id *cma_id);
> 
>

Because RDMA connections are established over a reliable connection
there is a distinct need to have some form of throttle control.
Excess connection requests received will be non-peer-rejected.
With kDAPL this is controlled by the size of the connection EVD.
When using a callback what is needed is for the callback function
to be able to report that it could not accept the connection request
object -- resulting in a non-peer reject rather than a peer reject.
To prevent DoS attacks it is essential that this be done as close
to the driver as possible.
 

> 
> /**
>  * ib_cma_get_src_ip - this function performs "rarp", asynchronicly
>  *   from cma_id to src ip
>  * @cma_id: the cma_id will have to include the path data received 
>  *   in the request handler
>  * @src_ip: source ip of the initiator
>  */
> int ib_cma_get_src_ip(union ib_cma_id *cma_id,
> 		      ib_cma_rarp_handler rarp_handler,
> 		      void *context);
> 

Or whatever equivalent logic needed to translate the 'cma_id' to the 
peer's 'src ip'. This is extracting a field for iWARP.