[ofw] choice between IBAL and winverbs

[Richard Frank]

We are investigating what is required to port the RDS driver to OFW..

RDS is a kernel mode driver which uses the CMA (rdmacm) interfaces.

A general question we are thinking about is whether or not there is a 
requirement for a kernel mode
(direct call) cma (rmda_cm) library / driver for use by kernel mode 
clients in Windows
and how to go about building it.

We are looking for input - to see if there is a general requirement for 
this kernel rdma_cm lib and how
to build it - or - if we should consider alternatives which leverage 
existing kernel interfaces / drivers.

Some alternatives are:

a) leverage existing wvdriver.sys - calling it directly from kernel 
(creating IRPs, etc) and
feeding it with required information to complete address translation, 
etc.. possibly using
kernel mode sockets interfaces..

b) somehow leveraging the existing user mode lib rdma cm via a user mode 
daemon which
handles requests to perform connection management operations from kernel 
mode drivers..

c) work directly with IBAL.. etc..

Smith, Stan wrote:
> ------------------------------------------------------------------------
> *From:* Sivaselvam Narayanasamy 
> [mailto:sivaselvam.narayanasamy at oracle.com]
> *Sent:* Tuesday, July 21, 2009 11:36 AM
> *To:* Smith, Stan
> *Cc:* Prasanna Vv; ofw at lists.openfabrics.org; Sivaselvam Narayanasamy
> *Subject:* RE: [ofw] choice between IBAL and winverbs
>
> Stan,
>  
> Prasanna's question may be :  From a kernel mode driver,  which is 
> the preferred route, to talk to the layers below :- 
>  
> a) through the winverbs.sys driver ( available in WinOF 2.1 )  or
> b) through the IBAL kernel driver
>  
> ? 
>  
> I'm not a Windows driver or winverbs expert, Sean and Fab are the guys 
> who wrote the code - guys please chime in.
>  
> Kernel mode IB verbs clients in the WinOF stack use a direct call 
> interface. The IPoIB driver requests an IBAL kernel interface based on 
> a well-known GUID. IBAL returns an interface struct populated at 
> runtime with IBAL function pointers from the ibal verbs driver (aka 
> ibbus.sys) memory image.
>  
> The ibbus.sys driver (an upper class filter driver for an InfiniBand 
> Controller (aka HCA)) creates a DOS file/ IOCTL() interface so 
> user-mode apps can access IBAL verbs functionality.
> The difference between user-mode IBAL and kernel mode IBAL is IOCTL() 
> vs. direct-call interfaces.
> Other considerations:
>    Marshalling of arguments
>    accessibility of args
>       inputs -[direct access or copyin()].
>       outputs -  [direct access or copyout()].
>  
> Winverbs does not export a kernel mode direct-call interface at this 
> time; winverbs only exports an IOCTL() interface accessible only from 
> user mode.
>  
> The  only kernel mode verbs interface choice at this time is to use IBAL.
>  
> Hopefully this clears up any confusion.
>  
> stan.
>  
> Thanks
> Siva
>
>     -----Original Message-----
>     *From:* ofw-bounces at lists.openfabrics.org
>     [mailto:ofw-bounces at lists.openfabrics.org]*On Behalf Of *Smith, Stan
>     *Sent:* Tuesday, July 21, 2009 10:52 PM
>     *To:* Prasanna Vv; ofw at lists.openfabrics.org
>     *Subject:* RE: [ofw] choice between IBAL and winverbs
>
>     ------------------------------------------------------------------------
>     *From:* ofw-bounces at lists.openfabrics.org
>     [mailto:ofw-bounces at lists.openfabrics.org] *On Behalf Of *Prasanna Vv
>     *Sent:* Tuesday, July 21, 2009 10:16 AM
>     *To:* ofw at lists.openfabrics.org
>     *Subject:* Re: [ofw] choice between IBAL and winverbs
>
>     Thanks Stan.
>     For kernel mode IB  interface,  whether  winverbs-driver IOCTL can
>     be used or IBAL is the only choice? 
>      
>     Unfortunately I was not clear w.r.t. user-mode or kernel mode POV
>     (Point-Of-View).
>     A kernel mode implementation of winverbs is not available today.
>     The work is being discussed and will be available in the WinOF 2.2
>     release (late Q4'09/Q1'10).
>     Kernel mode winverbs will not be available in the upcoming WinOF
>     2.1 release.
>      
>     stan. 
>
>     Regards,
>     Prasanna
>
>     Smith, Stan wrote:
>>     Prasanna Vv wrote:
>>       
>>>     Hello All,
>>>
>>>     In WINOF 2.1, most of the ULPs are based on Winverbs, while IPoIB
>>>     driver continues to use IBAL .
>>>         
>>
>>     Actually most of the ULPs are based on IBAL for historical reasons.
>>     It's only newer DAPL providers (Socket-CM & rdma-CM) plus OFED diags which are using OFED compatibility (aka libibverbs/winverbs).
>>     NetworkDirect continues to use the IBAL IF although there is work-in-progress to switch over to winverbs to take advantage of overlapped I/O ops.
>>
>>       
>>>     I wish to know if OFED will continue to support and enhance IBAL as
>>>     complimentary approach for Winverbs for IB verbs interface ?  Or IBAL
>>>     may be phased out ?
>>>         
>>
>>     The IBAL interface will be supported well into the future, measured in years.
>>     Under the covers, how the IBAL interface is implemented may change such that an IBAL function could transparently become a wrapper for a call into winverbs.
>>     The 'BIG" deal with IBAL is how it implements PNP and how IPoIB and drivers rely on this implementation.
>>     There are no current plans to change all drivers and IPoIB to use Microsoft PNP and remove this functionality from IBAL; although it would be an excellent idea (very large task). For the PNP reasons, IBAL will stick around for some time.
>>
>>     For future development, I would recommend the OFED compatibility layer (libibverbs/winverbs).
>>
>>     Stan.
>>
>>       
>>>     Thanks,
>>>     Prasanna
>>>
>>>     _______________________________________________
>>>     ofw mailing list
>>>     ofw at lists.openfabrics.org
>>>     http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>>>         
>>
>>       
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> ofw mailing list
> ofw at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw