[openib-general] Connection Manager Abstraction proposition - header file

[Caitlin Bestler]

On 8/24/05, Guy German <guyg at voltaire.com> wrote:
> 
> enum ib_cma_event {
>         IB_CMA_EVENT_ESTABLISHED,
>         IB_CMA_EVENT_REJECTED,
>         IB_CMA_EVENT_DISCONNECTED,
>         IB_CMA_EVENT_UNREACHABLE
> };
> 

The events need to distinquish between Rejected and Peer Rejected.
For example, a TCP connection could be rejected by the peer stack
for lack of capacity to relay the Connection Request to the peer for
approval. That is neither a peer rejection nor an "unreachable" event.
 
> enum ib_qos {
>         IB_QOS_BEST_EFFORT = 0,
>         IB_QOS_HIGH_THROUGHPUT = (1 << 0),
>         IB_QOS_LOW_LATENCY = (1 << 1),
>         IB_QOS_ECONOMY = (1 << 2),
>         IB_QOS_PREMIUM = (1 << 3)
> };
> 
> enum ib_connect_flags {
>         IB_CONNECT_DEFAULT_FLAG = 0x00,
>         IB_CONNECT_MULTIPATH_FLAG = 0x01
> };
> 
> /*
>  * for ib_cma_get_src_ip - ib_cma_id will have to include
>  * the path data received in the request handler
>  */
> union ib_cma_id{
>         struct ib_cm_id *cm_id;
>         u32 iwarp_id;
> };
> 
> typedef void (*ib_cma_rarp_handler)(struct sockaddr *src_ip, void *context);
> typedef void (*ib_cma_ac_handler)(enum ib_cma_event event, void *context);
> typedef void (*ib_cma_event_handler)(enum ib_cma_event event, void *context,
>                                      void *private_data);
> typedef void (*ib_cma_listen_handler)(union ib_cma_id *cma_id,
>                                       void *private_data, void *context);
> 
> struct ib_cma_conn {
>         struct ib_qp *qp;
>         struct ib_qp_attr *qp_attr;
>         struct sockaddr *dst_ip;
>         __be64 service_id;
>         void *context;
>         ib_cma_event_handler cma_event_handler;
>         const void *private_data;
>         u8 private_data_len;
>         u32 timeout;
>         enum ib_qos qos;
>         enum ib_connect_flags connect_flags;
> };
> 
> 
> /**
>  * ib_cma_get_device - Returns the device to be used according to
>  *   the destination ip address (this can be detemined according
>  *   to the local routing table). Call this function before
>  *   creating the qp. If using link-local IPv6 addresses
>  * @remote_address: The destination address for connection
>  * @device: The device to use (returned by the function)
>  */
> int ib_cma_get_device(struct sockaddr *remote_address,
>                       struct ib_device **device);
> 
> 

This would need\ to be based on the remote_address *and* CoS.
For example there could be two devices that reach the same
destination network, but with different speeds.

> /**
>  * ib_cma_connect - this is the connect request function, called by
>  *   the active side. The consumer registers an upcall that will be
>  *   initiated by the cma with an appropriate connection event
>  *   notification (established/rejected/disconnected etc)
>  * @cma_conn: This structure contains the following connection parameters:
>  *   @qp: qp for establishing the connection
>  *   @qp_attr: only relevant attributes are used
>  *   @dst_ip: destination ip address
>  *   @service_id: destination service id (port)
>  *   @context: context to be returned in the callback
>  *   @cma_event_handler: the upcall function for the active side
>  *   @private_data: private data to be received at the listener upcall
>  *   @private_data_len: private data length (max 255)
>  *   @timeout:
>  *   @qos: Quality os service for the rc
>  *   @connect_flags: default or multipath connection
>  * @cma_id: This returned handle is a union (different in ib and iwarp)
>  *   in ib - it is the cm_id.
>  */

As noted above, the QoS is also needed to even select the device.

> int ib_cma_connect(struct ib_cma_conn *cma_conn,
>                    union ib_cma_id *cma_id);
> 
> 
> /**
>  * ib_cma_disconnect - this function disconnects the rc. It can be
>  *   called, by either the passive or active side
>  * @qp: the connected qp to disconnect
>  * @cma_id: On the active side- this handle is the one returned
>  *   when ib_cma_connect was called.
>  *   On the passive side- this handle was accepted in cma_listen callback
>  */
> int ib_cma_disconnect(struct ib_qp *qp, union ib_cma_id *cma_id);
> 
> 
> /**
>  * ib_cma_sid_listen - this function is called by the passive side. It is
>  *   listening on a the specified port (ib service id) for incomming
>  *   connection requests
>  * @device: ? need to resolve this issue
>  * @service_id: service id (port) to listen on
>  * @context: user context to be returned in the callback
>  * @cm_listen_handler: the listen callback
>  * @cma_id: cma handle for the passive side
>  */
> int ib_cma_sid_listen(struct ib_device *device, __be64 service_id,
>                       void *context, ib_cma_listen_handler cm_listen_handler,
>                       union ib_cma_id *cma_id);
> 
> 

There could be an option to listen on a specific address as well.




When the current kDAPL interface is split up several state issues
need ot be explicitly dealt with since they are no longer inherited.

For iWARP these include how TCP layer errors are handled during
connection setup (non-peer reject being the most accurate description
that attempts to be transport neutral).

It also needs to be clear that while Connection Requestsw do not have
to be dealt with synchronously or even in-order, they are perishable goods
that must be dealt with promptly. The exact deadlines are transport
dependent, but should not be of concern to a typical application that
responds to a connection request in anything that could in good faith
be considered a prompt response. On the IB side that means some
retries may be implemented transparently, and that a consumer that
wanted explicit control over retries would need to use the current IB
specific API.