[openib-general] gen2/rnic-pi differences

[Caitlin Bestler]

The wr_id field is the "minimal support" that I was
referring to (that and the context on callbacks).

DAPL and IT-API really need more than that to optimally
implement features such as:
	storing the ULP's 64-bit cookie.

	controlling "notification status" on a per-DTO level
	rather than just always leaving the CQ on a "next event"
	basis.

	implementing graceful disconnect (disconnect after
	the last send request completes).

	allowing the ULP to flush a request through as
	a marker.

	returning the EP or RMR pointer with the completion.

The 64 bit wr_id is fully consumed just implementing
the first option.

Without further support the Access Layer (DAT/IT-API/whatever)
must create its own parallel data structure to shadow the
work request (a DTO_COOKIE in the reference implementation).

Such a structure has been found in every DAT implementation
that I am aware of to date for the simple reason that the
verb layers have not offered much support beyond the 64-bit
work request id and a context field on callbacks.

But it is in fact very easy to solve all of these problems
at the verb layer, even if there is no hardware support.

RNIC-PI defined several verb layer features that if supported
eliminate the need for a DTO_COOKIE. If the information can
be integrated with existing verb layer structures it is a 
major improvement in efficiency, at worst case it merely
requires the verb layer to implement the same workarounds
that the Access Layer is already forced to use.

These features are:
	all verb layer objects have a consumer supplied 
	identifier (os_data) that is used to identify that
	object back to the consumer in all completions and
	callbacks. So instead of getting the QPID you get
	the EP pointer (assuming that is what you supplied).

	Three flags are identified per work request that can
	be ignored, passed through or fully implemented. They
	are Local Solicited, Consumer0 and Consumer1. If 'Local
	Solicited' is defined it means that completion of the
	work request should be treated as though it were a
	solicited event (i.e., it qualifies for 'next solicited
	event' callback notification).

with these changes there is no need for a DTO_COOKIE, which
reduces the number of memory touches, simplifies code, and
may eliminate associated locking to protect the DTO_COOKIE
resource itself.


	
	 

> -----Original Message-----
> From: James Lentini [mailto:jlentini at netapp.com] 
> Sent: Thursday, June 30, 2005 3:18 PM
> To: Caitlin Bestler
> Cc: openib-general; rdma-developers at lists.sourceforge.net
> Subject: Re: [openib-general] gen2/rnic-pi differences
> 
> 
> 
> On Thu, 30 Jun 2005, Caitlin Bestler wrote:
> <snip>
> > os_data / Identification of Consumer Objects
> >
> > 	gen2 provides minimal support for identification of RDMA
> > 	resources using consumer supplied handles. A user-supplied
> > 	context is available in callbacks, but not in work completions.
> 
> Gen2 work completions store a user-supplied context in the 
> ib_wc structure's wr_id field.
>