[ofw] port RDS to OFW - request for input...

[Sean Hefty]

>Taking Sean's divide-n-conquer approach let's talk about kernel shim layers.
>
>1) How hard/reasonable would it be to provide an ibverbs interface layer over
>IBAL verbs; given it's RDS specific and targeted at data transfer (minimal
>subset)?
>
>2) Is it possible/reasonable to construct an rdma_cm interface layer over IBAL-
>CM? How different are the semantics?

The question to me is should Linux *kernel* shim layers be supported by WinOF
between drivers.  My initial reaction to that is no.  Linux kernel interfaces
are not stable and aren't the same between OFED and kernel.org at this point.
Instead, I would focus on whether WinOF provides similar kernel functionality as
OFED.  (Don't forget that IBAL was derived from a Linux implementation, which is
why the PnP is such a mess when integrated into the Windows PnP environment.)
Of course, that doesn't mean that we need to invent differences between the
platforms if we can make due without any.  And it may be that a shim is so
trivial that it's easy to do.

As for what it would take to create a Linux verbs shim, that shouldn't be too
difficult over any verbs flavor.  And I'm guessing the costs would be about the
same.  I would look at going directly to the HCA driver.  So, the shim would
either go to the HCA driver, or the winverbs kernel interface would _be_ the
shim.  (That's basically all the kernel driver 'verbs' portion does today,
beyond protecting the kernel from misbehaving user space apps.)

For the rdma_cm functionality, see my other post for the problem: there's not an
easy out for handling address resolution.  The rdma_cm is a consumer of the IB
CM, so could use the IbaCm interfaces (see ib_cm_ifc.h), which is separate from
the 'IBAL' interface (ib_al_ifc.h), but still exported by the ibal.sys driver.

>Question:  Are #1 & #2 any easier, faster or more sane than building kernel
>mode winverbs interfaces for FMR, rdma_cm and ibverbs-data-xfer?

Combined, the winverbs kernel driver and user space library are just shy of 9000
lines of code.  A kernel winverbs interface would likely be less than that, even
if FMR were added.  That's probably a good start for forming an estimate.  As
Tzachi mentioned, the rdma_cm portion of that is about 1000 lines.

>Building winverbs interfaces:
>  FMR and ibverbs prototype development: 1 head 12 weeks; kind of hard to test
>without a CM?
>  The 'real' issue is rdma_cm - building all the QP 0+1 MAD connection mgmt
>handing and speaking of MAD handling, IBAL, winverbs; who is in charge of MAD
>processing?  SWAG - 1 head 24 weeks to a working prototype.

One would want to start with the ib_cm_ifc.h interface and build from there.
You may be able to copy/paste/modify/factor out common code/merge the wv_ep.c
file to create a kernel rdma_cm interface.

>Shimming IBAL -
>  FMR and ibverbs interfaces: 1 head 8 weeks
>  rdma_cm IF over IBAL-CM: 1 head 12 weeks?

If WinOF is going to absorb the cost of creating new kernel interfaces, I would
rather see the cost going to a longer term windows verbs solution.  Of course,
this may mean that RDS needs to provide a shim to the new interface, rather than
creating a shim to an existing interface.

>Sean, would you like to explain how whacked all this is?  :-)

Yes - there's only one of each of us.  :)