[openib-general] GSI compromise

[Sean Hefty]

There are still some disagreements on the GSI APIs and overal model, and I would like to try to get everyone together on this.  Using the gsi.h file under trunk/contrib/voltaire/access as a base, I tried to modify the API slightly to meet some sort of compromise.  The modified file is below.  Some comments about the changes:

* I tried to make the API for QP0 and QP1, but I focused on QP1.  "gsi" was replaced with "mad" because of this.
* I added struct ib_mad_reg to replace the handle that is currently being returned.
* I combined ib_gsi_buffer and ib_gsi_msg.  (I don't think there's any reason to have the ib_gsi_msg have a list.  I know that this was done in the Intel gen1 code, but I think it was the wrong approach to take.  It's *much* simpler for clients to send/receive a single data buffer.  In fact, I'd be surprised if there was any client that didn't copy the chained buffers into a single buffer.)
* I added send_context and timeout fields to the gsi_msg.  These are intended to be set when receiving a response to an outgoing request.
* I changed the registration call to operate based on method values.
* I removed the gsi_redir_class call for now, since there's some disagreement on how this should best be accomplished.  After looking at the struct ib_gsi_reg, I have started thinking of some ideas on how this could work.

What is still missing is information about RMPP.  My preference would be for the clients to indicate to the access layer which classes/methods require RMPP, rather than hard-coding it directly into the access layer.  This allows a single implementation of RMPP, but gives the clients control over when it should be invoked.

For redirection, if you examine struct ib_gsi_reg, there's a pointer to a qp.  For normal registration calls, I'm expecting that this value will go directly to QP1.  For QP redirection, we can add a calls such as:

struct ib_mad_reg *ib_qp_redir(struct ib_qp *qp,
			       ib_mad_send_handler send_handler,
			       ib_mad_recv_handler recv_handler,
			       void *context);

int ib_process_wc(struct ib_mad_reg *mad_reg,
		  struct ib_wc *wc);

The client would allocate and manage the QP and CQ(s).  The redirect call simply informs the access layer that RMPP and request/response services should be enabled on that QP.  When the client removes a work completion for that QP, it hands the work completion to the access layer for processing.  The access layer can then perform RMPP and request/response matching.

These calls wouldn't be needed for something like the CM, but would be helpful for an SA that required redirection, but didn't want to re-implement RMPP.  I think that given these calls it would be possible to implement a "redirection layer" similar to the one mentioned in the currently proposed GSI.  And I'm guessing that you could shim the proposed GSI over this API without too much trouble, especially if it disabled RMPP.

Please respond with comments.  Again, I'm hoping that we can reach a set of APIs that everyone is comfortable with.

- Sean
/*
  This software is available to you under a choice of one of two
  licenses.  You may choose to be licensed under the terms of the GNU
  General Public License (GPL) Version 2, available at
  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
  license, available in the LICENSE.TXT file accompanying this
  software.  These details are also available at
  <http://openib.org/license.html>.

  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  SOFTWARE.

  Copyright (c) 2004 Mellanox Technologies Ltd.  All rights reserved.
  Copyright (c) 2004 Infinicon Corporation.  All rights reserved.
  Copyright (c) 2004 Intel Corporation.  All rights reserved.
  Copyright (c) 2004 Topspin Corporation.  All rights reserved.
  Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
*/

#if !defined( IB_MAD_H )
#define IB_MAD_H

#include "ib_verbs.h"

typedef void (*ib_mad_send_handler)(struct ib_mad_reg *mad_reg,
				    struct ib_mad_msg *msg);
typedef void (*ib_mad_recv_handler)(struct ib_mad_reg *mad_reg,
				    struct ib_mad_msg *msg);

struct ib_mad_reg {
	struct ib_device	*device;
	struct ib_qp		*qp;
	ib_mad_recv_handler	recv_handler;
	ib_mad_send_handler	send_handler;
	void			*context;
};

struct ib_mad_msg {
	struct ib_mad_msg	*next;

	void			*buf;
	int			length;

	/* send_context is set on a receive to context on matching send */
	void			*send_context;
	void			*context;
	int			timeout_ms;

	enum ib_wc_status_t	status;

	u32			remote_qp;
	u32			remote_q_key;
	u16			remote_lid;
	u16			pkey_index;
	u8			service_level;
	u8			path_bits;
	u8			global_route;
};

/*
 * mgmt_class, mgmt_class_version, and method_array are only
 * required if the user wishes to receive unsolicited MADs
 */
struct ib_mad_reg *ib_mad_reg_class(struct ib_device *device,
				    u8 port,
				    enum ib_qp_type qp_type,
				    u8 mgmt_class,
				    u8 mgmt_class_version,
				    u8 *method_array, 
				    ib_mad_send_handler send_handler,
				    ib_mad_recv_handler recv_handler,
				    void *context);

int ib_mad_dereg_class(struct ib_mad_reg *mad_reg);

int ib_mad_post_send_msg(struct ib_mad_reg *mad_reg,
			 struct ib_mad_msg *msg);

/* Proposed redirection support. */
struct ib_mad_reg *ib_qp_redir(struct ib_qp *qp,
			       ib_mad_send_handler send_handler,
			       ib_mad_recv_handler recv_handler,
			       void *context);

ib_process_wc(struct ib_mad_reg *mad_reg,
	      struct ib_wc *wc);

#endif /* IB_MAD_H */