[openib-general] Connection Manager Abstraction proposition -header file
Guy German
guyg at voltaire.com
Wed Aug 24 08:04:38 PDT 2005
Hi Tom,
On Wed, 2005-08-24 at 11:07 -0400, Tom Tucker wrote:
> Guy:
>
>
> I think we're on the right track. Regarding the 'get-device' function,
> the ib_at_route_by_ip service already returns a data structure that
> includes the device pointer. This data structure also includes the local
>
> and next hop remote IP addresses which is a good thing. First, it avoids
>
> requiring the connect method to look this data up again. For IB, these
> IP addresses are first converted to GID/LID with AT and then to a path
> record. For iWARP, these addresses are fine as is, and avoid having the
> connect method do a second lookup to find the next hop given the remote
> ip address. In other words, pass the ib_at_ib_route structure to the
> connect method along with the remote IP address to allow this info to
> be reused.
You are right about the functionality of ib_at_route_by_ip, but I don't
think you want to implement at.c for iwarp. My thoughts were that one
abstraction layer is enough. In ib it can be mapped to at module calls,
and in iwarp to a different implementation.
> With regard to ib_cma_id, what is the iwarp_id? Is this the remote port?
You know iwarp needs better then me, it's the information you need to
keep in order to open and close the connection.
>
> With regard to listen, it should not require a device pointer because
> the
> app may in fact be listening on multiple devices. Also, the service_id
> does not provide enough information to determine which devices to listen
> on. It should be local ip (could be 0 for wild-card), and local port
> (service id).
I agree that this part needs to be more thought of. If you look at the
documentation - I left a "?" near ib_device.
>
> The listen callback function needs to know the device on which the
> connection request was received. This could be included in the
> ib_cma_id structure.
Yes. This is part of the same issue as the previous one.
Thanks,
Guy.
>
> > -----Original Message-----
> > From: openib-general-bounces at openib.org
> > [mailto:openib-general-bounces at openib.org] On Behalf Of Guy German
> > Sent: Wednesday, August 24, 2005 7:21 AM
> > To: openib-general at openib.org
> > Subject: [openib-general] Connection Manager Abstraction
> > proposition -header file
> >
> > /*
> > * Copyright (c) 2005 Voltaire Inc. All rights reserved.
> > *
> > * This Software is licensed under one of the following licenses:
> > *
> > * 1) under the terms of the "Common Public License 1.0" a
> > copy of which is
> > * available from the Open Source Initiative, see
> > * http://www.opensource.org/licenses/cpl.php.
> > *
> > * 2) under the terms of the "The BSD License" a copy of which is
> > * available from the Open Source Initiative, see
> > * http://www.opensource.org/licenses/bsd-license.php.
> > *
> > * 3) under the terms of the "GNU General Public License
> > (GPL) Version 2" a
> > * copy of which is available from the Open Source Initiative, see
> > * http://www.opensource.org/licenses/gpl-license.php.
> > *
> > * Licensee has the right to choose one of the above licenses.
> > *
> > * Redistributions of source code must retain the above copyright
> > * notice and one of the license notices.
> > *
> > * Redistributions in binary form must reproduce both the
> > above copyright
> > * notice, one of the license notices in the documentation
> > * and/or other materials provided with the distribution.
> > *
> > */
> >
> > /*
> > * This header file as a preliminary proposition for a
> > connection manager
> > * abstraction layer (cma) for IB and iwarp
> > * - there is an assumption that iwarp uses the same openib
> > qp terminology in
> > * the rest of the verbs, and the only place needs
> > abstraction is the cm.
> > * - This proposition assumes that the address translation
> > is done in the cma
> > * layer.
> > * - The cma also modifies the qp states to init/rtr/rts and
> > error as needed.
> > * - for calling accept/reject or disconnect on the passive
> > side you need to
> > * use the cma handle accepted in ib_cma_listen cb.
> > * - cma_id is created when calling connect or listen and
> > destroyed when
> > * accepting disconnected/rejected/unreachable events on
> > either active
> > * side (connect cb) or passive side (accept cb)
> > */
> >
> > #ifndef IB_CMA_H
> > #define IB_CMA_H
> >
> > #include <linux/socket.h>
> >
> > enum ib_cma_event {
> > IB_CMA_EVENT_ESTABLISHED,
> > IB_CMA_EVENT_REJECTED,
> > IB_CMA_EVENT_DISCONNECTED,
> > IB_CMA_EVENT_UNREACHABLE
> > };
> >
> > enum ib_qos {
> > IB_QOS_BEST_EFFORT = 0,
> > IB_QOS_HIGH_THROUGHPUT = (1 << 0),
> > IB_QOS_LOW_LATENCY = (1 << 1),
> > IB_QOS_ECONOMY = (1 << 2),
> > IB_QOS_PREMIUM = (1 << 3)
> > };
> >
> > enum ib_connect_flags {
> > IB_CONNECT_DEFAULT_FLAG = 0x00,
> > IB_CONNECT_MULTIPATH_FLAG = 0x01
> > };
> >
> > /*
> > * for ib_cma_get_src_ip - ib_cma_id will have to include
> > * the path data received in the request handler
> > */
> > union ib_cma_id{
> > struct ib_cm_id *cm_id;
> > u32 iwarp_id;
> > };
> >
> > typedef void (*ib_cma_rarp_handler)(struct sockaddr *src_ip,
> > void *context);
> > typedef void (*ib_cma_ac_handler)(enum ib_cma_event event,
> > void *context);
> > typedef void (*ib_cma_event_handler)(enum ib_cma_event event,
> > void *context,
> > void *private_data);
> > typedef void (*ib_cma_listen_handler)(union ib_cma_id *cma_id,
> > void *private_data, void
> > *context);
> >
> > struct ib_cma_conn {
> > struct ib_qp *qp;
> > struct ib_qp_attr *qp_attr;
> > struct sockaddr *dst_ip;
> > __be64 service_id;
> > void *context;
> > ib_cma_event_handler cma_event_handler;
> > const void *private_data;
> > u8 private_data_len;
> > u32 timeout;
> > enum ib_qos qos;
> > enum ib_connect_flags connect_flags;
> > };
> >
> >
> > /**
> > * ib_cma_get_device - Returns the device to be used according to
> > * the destination ip address (this can be detemined according
> > * to the local routing table). Call this function before
> > * creating the qp. If using link-local IPv6 addresses
> > * @remote_address: The destination address for connection
> > * @device: The device to use (returned by the function)
> > */
> > int ib_cma_get_device(struct sockaddr *remote_address,
> > struct ib_device **device);
> >
> >
> > /**
> > * ib_cma_connect - this is the connect request function, called by
> > * the active side. The consumer registers an upcall that will be
> > * initiated by the cma with an appropriate connection event
> > * notification (established/rejected/disconnected etc)
> > * @cma_conn: This structure contains the following
> > connection parameters:
> > * @qp: qp for establishing the connection
> > * @qp_attr: only relevant attributes are used
> > * @dst_ip: destination ip address
> > * @service_id: destination service id (port)
> > * @context: context to be returned in the callback
> > * @cma_event_handler: the upcall function for the active side
> > * @private_data: private data to be received at the listener upcall
> > * @private_data_len: private data length (max 255)
> > * @timeout:
> > * @qos: Quality os service for the rc
> > * @connect_flags: default or multipath connection
> > * @cma_id: This returned handle is a union (different in ib
> > and iwarp)
> > * in ib - it is the cm_id.
> > */
> > int ib_cma_connect(struct ib_cma_conn *cma_conn,
> > union ib_cma_id *cma_id);
> >
> >
> > /**
> > * ib_cma_disconnect - this function disconnects the rc. It can be
> > * called, by either the passive or active side
> > * @qp: the connected qp to disconnect
> > * @cma_id: On the active side- this handle is the one returned
> > * when ib_cma_connect was called.
> > * On the passive side- this handle was accepted in
> > cma_listen callback
> > */
> > int ib_cma_disconnect(struct ib_qp *qp, union ib_cma_id *cma_id);
> >
> >
> > /**
> > * ib_cma_sid_listen - this function is called by the passive
> > side. It is
> > * listening on a the specified port (ib service id) for incomming
> > * connection requests
> > * @device: ? need to resolve this issue
> > * @service_id: service id (port) to listen on
> > * @context: user context to be returned in the callback
> > * @cm_listen_handler: the listen callback
> > * @cma_id: cma handle for the passive side
> > */
> > int ib_cma_sid_listen(struct ib_device *device, __be64 service_id,
> > void *context, ib_cma_listen_handler
> > cm_listen_handler,
> > union ib_cma_id *cma_id);
> >
> >
> > /**
> > * ib_cma_sid_destroy - this functionis is called on the
> > passive side, to
> > * stop listenning on a certain sevice id
> > * @cma_id: the same cma handle received when
> > ib_cma_sid_listen was called
> > */
> > int ib_cma_sid_destroy(union ib_cma_id *cma_id);
> >
> >
> > /**
> > * ib_cma_accept - call on the passive side to accept a
> > connection request
> > * @cma_id: this handle was accepted in cma_listen callback
> > * @qp: the connection's qp
> > * @private_data: private data to send back to the initiator
> > * @private_data_len: private data length
> > * @context: user context to be returned in the callback
> > * @cm_accept_handler: the cma accept callback - triggered
> > when RTU ack
> > * received
> > */
> > int ib_cma_accept(union ib_cma_id *cma_id, struct ib_qp *qp,
> > const void *private_data, u8 private_data_len,
> > void *context, ib_cma_ac_handler cm_accept_handler);
> >
> > /**
> > * ib_cma_reject - call on the passive side to reject a
> > connection request.
> > * This call destroys the cma_id, hence when the active side accepts
> > * the reject the cma_id is already destroyed.
> > * @cma_id: this handle was accepted in cma_listen callback
> > * @private_data: private data to send back to the initiator
> > * @private_data_len: private data length
> > */
> > int ib_cma_reject(union ib_cma_id *cma_id, const void *private_data,
> > u8 private_data_len);
> >
> >
> > /**
> > * ib_cma_get_src_ip - this function performs "rarp", asynchronicly
> > * from cma_id to src ip
> > * @cma_id: the cma_id will have to include the path data received
> > * in the request handler
> > * @src_ip: source ip of the initiator
> > */
> > int ib_cma_get_src_ip(union ib_cma_id *cma_id,
> > ib_cma_rarp_handler rarp_handler,
> > void *context);
> >
> > #endif /* IB_CMA_H */
> >
> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org
> > http://openib.org/mailman/listinfo/openib-general
> >
> > To unsubscribe, please visit
> > http://openib.org/mailman/listinfo/openib-general
> >
More information about the general
mailing list