[Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux

[Caitlin Bestler]

> -----Original Message-----
> From: Grant Grundler [mailto:iod00d at hp.com] 
> Sent: Friday, May 27, 2005 7:15 PM
> To: Caitlin Bestler
> Cc: Bob Woodruff; Roland Dreier; James Lentini; 
> Arkady.Kanevsky at netapp.com; Venkata Jagana; 
> rdma-developers at lists.sourceforge.net; openib-general at openib.org
> Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB 
> and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
> 
> On Fri, May 27, 2005 at 06:24:46PM -0700, Caitlin Bestler wrote:
> > Oh that's a great development plan for me to propose to my business 
> > people. Let's go write lots of code, and *after* we write it the IB 
> > people (our competitors) have agreed to *look* at it (but 
> they're all 
> > busy, don't
> > forget) and *then* we just might get merged in.
> 
> You will never get a guarantee your code will go into 
> kernel.org regardless of which path you take.
> I've seen three of the lead developers (a) promise to review 
> any proposed code changes and (b) indicate they are willing 
> to merge in support for RNICs.
> 

A commitment that after merger that the API will be transport
neutral is essential. That is, IB HCAs do not have to emulate
iWARP. iWARP HCAs do not have to emulate IB.

I fail to see why an advance agreement about the goal cannot
be reached. The iWARP vendors are willing to concede that
current Gen2 verb modules still have to be supported in a
merged solution.

What is your definiton of a merged solution? Is there a commitment
that "RDMA Services" does not mean "InfiniBand, and the iWARP
vendors have to pretend to be InfiniBand but we'll add one or
two fields for problems they *can't* solve."


> 
> Look at the relevant openib.org header files (e.g. ib_verbs.h,
> ib_user_verbs.h) and make a list of changes needed for it to 
> be useful to an RNIC.  Just pointing at RNIC-PI isn't interesting.
>

Well since we are talking about what a final transport neutral
API would look like, as a guide when writing new code that will
be compatible with the final merger, then looking at an example
of an API that attempts to be transport neutral strikes me as
highly relevant.

Is there anything in the general structure of RNIC-PI that you
do not like? The IB-specific portions are different than Gen2,
and that certainly would not be true of any final merged verbs.
So far the only comments received are that RNIC-PI was not written
to be inside the Linux Kernel. That's correct, it was designed to
be an OS neutral API that could be supported on a wide variety
of Operating systems. Changes *have* been made, such as always
having struct/enum/union tags, precisely to guarantee that a 
Linux version of rnicpi.h can be written that will be interoperable
with code written to the man pages.

As for gen2, here's one minor example:

struct ib_qp_attr {
	enum ib_qp_state	qp_state;		// States
defined are IB specific
	enum ib_qp_state	cur_qp_state;
	enum ib_mtu		path_mtu;		// The path MTU
is not a QP attribute in iWARP,
							// The LLP
maintains that. The maximum DDP Segment
							// size is
exported to the DDP layer, but not to 
							// the consumer.
	enum ib_mig_state	path_mig_state;	// Path migration is
handled at L2 or L3, it is 
							// not visisble
to an iWARP QP.
	u32			qkey;			// There is no
qkey
	u32			rq_psn;		// There is no psn.
There is an MSN for send/recvs.
							// It is not
normally exported to the consumer.
	u32			sq_psn;		// There is no psn.
	u32			dest_qp_num;	// The destination QP is
not known.
	int			qp_access_flags;  // presumably mostly
compatible, but its not documented
							// in the .h
file	
	struct ib_qp_cap	cap;			// different
capacities are controllable between
							// iWARP and IB.
RNIC-PI defined the union of the
							// two, and left
it to the providers to simply replicate
							// counts that
are redundant under their transport.
							// Example:
RDMAC allows different sge capacities
							// depending on
the type of work request (send vs. write).
	struct ib_ah_attr	ah_attr;		// entirely IB
specific information. TCP equivalents are
							// not part of
QP state.
	struct ib_ah_attr	alt_ah_attr;	// see ah_attr
	u16			pkey_index;		// VLANs are the
equivalent, and are managed below the
							// RDMA layer
	u16			alt_pkey_index;	// ditto
	u8			en_sqd_async_notify;	// There is no
sqd state
	u8			sq_draining;
	u8			max_rd_atomic;		// there are no
atomics
	u8			max_dest_rd_atomic;	
	u8			min_rnr_timer;		// Retries and
pauses are handled below the RDMA layer
	u8			port_num;			//
Egress ports are managed at the IP layer, not as 
								// part
of the RDMA layer. Egress ports can be 
								//
reassigned without changing the QP's state
								// at
the RDMA layer
	u8			timeout;			//
managed by the LLP (TCP or SCTP)
	u8			retry_cnt;			//
managed by the LLP (TCP or SCTP)
	u8			rnr_retry;			//
managed by the LLP (TCP or SCTP)
	u8			alt_port_num;		// see port num
	u8			alt_timeout;		// see timeout
};

 

In a merged transport neutral API virtually the entire current struct
would
become part of an ib union. Just adding comments that this field is "IB
only"
doesn't achieve neutrality, it makes iWARP developers wade through
structs
to determine which fields can be effected, and makes it all too likely
that
an application developer will set a field not realizing that an iWARP
RNIC
will not look at that field.


If a transport neutral API based on unioning everything that is not
transport
neutral is not acceptable as the goal of eventual merger is not
acceptable
then the emphasis should be on kDAPL as the transport neutral API.