[openib-general] comments on DAT registry in OpenIB

[Kanevsky, Arkady]

Dear DAT and OpenIB members,
There is a debate going on on OpenIB and DAT reflectors which is going
around about kDAT registry for Linux.
 
I would like to review the requirements we had agreed at DAT
collaborative and captured in the kDAT
and uDAT specs and review DAT registry in OpenIB from that prospective.
 
 
I would like to make it clear that I do NOT speak on behalf of DAT
Collaborative but
as just one of its members.
 

1.	Ability of Consumers to open IA based on its name 

	*	OpenIB supports it

2.	Support for Consumers to get a list of available IAs to open 

	*	OpenIB kDAT and uDAT registry provide this 
	*	OpenIB kDAT registry no longer provide Provider
attributes as stated above
	*	Preserves dat_registry_list_providers
	*	for kDAPL OpenIB changed dat_provider_info format so
binary compatibility not preserved, but source compatibility is
preserved.

		*	dat_provider_info differs between uDAPL and
kDAPL in OpenIB

	
3.	Ability to enumerate available IAs and their attributes 

	*	OpenIB supports that for uDAPL unchanged from DAPL SF RI

	*	for kDAPL openIB supports a single type of thread-safety
defined by the Linux kernel and the version of Linux kernel defines the
kDAPL APIs that kernel version supports.

		*	for kDAPL query will not return DAT version and
thread safety Provider attribute.

4.	Map IA_name to Provider library (kDAPL or uDAPL) 

	*	OpenIB kDAPL and uDAPL support this

5.	Ability for DAT providers to dynamically register and deregister
DAPL Provider 

	*	OpenIB supports that for kDAPL and uDAPL
	*	All existing registry APIs at DAPL SF RI are preserved

6.	Single static DAT registry - platform specific 

	*	kDAT and uDAT specs explicitly state that the DAT
registry is defined by the platform and DAT collaborative provided an
example of Registry for Linux and Windows and agree that the DAT
provided registry should be used by all providers. This ensures that DAT
Registry will support all Providers and DAT registry from one vendor
does not block other providers. 
	*	OpenIB kDAT registry is the Linux platform DAT registry
which achieves the goal of supporting all kDAPL providers. It also
provides additional benefit that it is Linux core which maintain kDAT
registry  instead of DAT Collaborative 
	*	OpenIB uDAT registry remains the DAT collaborative one
unchanged. 
	*	We can discuss whether or not we want to get uDAT
registry closer to the OpenIB kDAT one 
	*	The DAT registry for kDAPL and uDAPL are different at
DAPL SF RI and OpenIB maintains it.
	*	Some changes may be needed for kDAPL Registry hot plug
support for OpenIB. How it may impact uDAPL registry.

7.	ia_name is under system admin control 

	*	remains the same following a platform convention

8.	IA can represent 

		1.	single port 
		2.	several ports 
		3.	several HBAs or RNICs 
		4.	multiple IAs represent the same port

	*	OpenIB kDAPL currently implements #1. Members can submit
code patches to support other choices  
	*	OpenIB uDAPL remains the same with current
implementation providing #1 under Provider control.

9.	Support for Consumers to get a list of available IAs to open 

	*	OpenIB kDAT and uDAT registry provide this 
	*	OpenIB kDAT registry no longer provide Provider
attributes as stated above

10.	DAT registry supports loading multiple DAPL Providers into the
same address space. 

		1.	A Provider library loaded into an address space
once 
		2.	A Provider library unloaded only when all open
instances of its IAs are closed 
		3.	The same Provider library can be loaded into
multiple address spaces

	*	OpenIB uDAPL continues to provide it 
	*	OpenIB kDAPL supports it

11.	DAT registry shall support polymorphism (Provider independency) 

		1.	Consumer call DAT functions by the DAT handle
independently from Provider is used 
		2.	DAT registry provides redirection 
		3.	dat_ia_open is Provider specific and sets up
redirection table per address space per Provider

			*	first time open ensures that table
redirection for a Provider is set up

	*	OpenIB kDAT and uDAT registry provide that 

		*	OpenIB kDAT registry preserves the DAT
redirection table as defined by DAT Collaborative 
		*	OpenIB kDAT registry preserves DAT_provider
structure 

			*	need to file errata to DAT to move
dat_ia_close after dat_ia_query to match DAPL SF RI and OpenIB one for
kDAPL and uDAPL

12.	The DAT_handle structure first field provides a pointer for
redirection 

	*	OpenIB kDAT and uDAT registry support this

		*	DAT registry interpret the DAT_handle as a
pointer to redirection table

13.	DAT registry supports multiple dat_ia_open for the same library
from the same and different address spaces 

	*	OpenIB DAT registry preserves that 
	*	OpenIB DAT registry supports refcount so the library is
not closed until the last Consumer leaves and library is loaded into an
address space once.

 So for uDAT Consumer and uDAT Provider which are compliant with uDAT
spec continue to work as before from DAPL SF RI.
 
kDAT Consumer no longer can choose Provider by DAT version # and thread
safety. The Linux kernel version defines which version of DAT and its
thread safety. Since kernel ULP is specific to the kernel configuration
this is OK.
If kernel module was choosing DAT Provider based on these attributes it
is no longer needed. The kernel module MUST match what kernel
configuration, including version number, provides. If kernel ULP behaves
differently based on version of DAT API and Provider thread safety this
switch will have to be replaced by kernel configuration/version switch.
This is "standard" for kernel ULP modules.
 
kDAT Provider will be effected. The dat_info structure is changed. It no
longer has fields for DAT major and minor versions and no field for
thread safety. DAT Provider must match kernel configuration/version DAT
API and thread-safety.
 
Others: kDAT Providers will need to maintain a separate tree for Linux
than for generic DAT or DAT for other platforms with appropriate Linux
kernel versioning/configuration support. For IB Providers it is not
needed since OpenIB kDAT provides that.  For iWARP vendors if they do
not plug-in into gen2 as HW driver then they need to match the
OpenIB/Linux kDAPL version DAT APIs and threading.
 
DAT Provider registration is different for uDAPL and kDAPL. This is true
at DAPL SF RI. And it is still the case for OpenIB.
 
The biggest difference from DAT Spec Linux Registry example and OpenIB
kDAT registry is a uDAPL/KDAPL registry shared configuration file that
contains all available Providers information. But this is just an
example. How Registry keeps the Provider info is up to a platform.
Consumer can not rely on the back door access to the static DAT registry
database. They should use the dat_registry_list_providers to get that
info. OpenIB kDAT registry preserves that API with a format change for
dat_info structure usable by Consumers.
 
uDAT and kDAT Providers code sharing is slightly diminished.
 
In summary, kDAT registry in OpenIB fulfils all DAT requirements.
There is a small change in one of the structure which impacts kDAPL
Consumers but is consistent with the way Linux kernel ULP modules
operation. The biggest change for DAT Collaborative is that this body no
longer "the sole Provider of DAT Registry". I view that as good news
since Linux maintainers are much better source of kDAPL and uDAPL
registry then DAT Collaborative.
 

Arkady Kanevsky                       email: arkady at netapp.com

Network Appliance                     phone: 781-768-5395

375 Totten Pond Rd.                  Fax: 781-895-1195

Waltham, MA 02451-2010          central phone: 781-768-5300

 

 
 
 

Arkady Kanevsky                       email: arkady at netapp.com

Network Appliance                     phone: 781-768-5395

375 Totten Pond Rd.                  Fax: 781-895-1195

Waltham, MA 02451-2010          central phone: 781-768-5300

 

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050629/b76f45b2/attachment.html>