[ofw] port RDS to OFW - request for input...

[Richard Frank]

Sean Hefty wrote:
>> Implement the entire protocol in user mode.
>>
>> There are a few advantages to this approach:
>> 1) From the existing libraries perspective one can use ND and RDMA cm.
>> 2) From the performance perspective this has advantage when it comes to
>> packet rate and latency (this is because one can send data and receive
>> it without going into the kernel). Achieving maximum BW should also not
>> be a problem with this approach. Handling interrupts should have
>> somewhat higher overhead.
>>     
>
> I thought about this option.  If Oracle uses one process with multiple threads
> on Windows, rather than using separate processes, this might work well.  The
> Windows port of libibverbs and librdmacm aren't that far off from the Linux
> kernel interfaces for verbs and the rdma_cm.  Porting the Linux kernel RDS
> driver to a Windows userspace library might not be that hard...?
>   
Yes - we too are looking at this approach.. and it is true that Oracle 
on Windows is a
multi-threaded process. So we could share a single connection across all 
threads in a
process and multiplex over it..

Some of the issues we identified are:

a) we need to keep the IPC behavior the same on all platforms - as we 
have the same IPC
clients running and inter-operating over them all - if we can do this 
while moving to user
mode that would work..

b) dynmaic registration of local buffers on the fly - requires a kernel 
mode transition. It is likely
that we'd be doing this given the nature of our IPC clients. So any 
performance advantages of
user mode may be lost..

c) RDS bcopy send operations complete immediately - e.g. are synchronous 
from the RDS client
perspective. The send buffer ownership is returned to the client as soon 
as it is copied to a kernel buffer..
Our RDS clients depend on this behavior.. If we move to user mode and 
post sends directly -
then we'd need to deal with async send completions which introduces 
latencies for send completions
to the local client.. etc..

d) we'd basically be doing a new RDS driver on windows - which we'd like 
to not do - but this is
a possibility.

> Another alternative that occurred to me was to integrate the RDS code directly
> into the winverbs driver, exposed through a new endpoint type or something like
> that.  I haven't given this idea much thought - just trying to brainstorm what
> options are available, and is why I wanted to know how Oracle interfaces to the
> RDS code.
>
>   
>> For the data path there is no choice but to use IBAL (currently there
>> are no winverbs in the user).
>>     
>
> It could speak directly to the HCA drivers for data path operations.
>
> Yet another option for RDS is to separate out the RDS protocol (use of verbs)
> from the connection establishment (use of the rdma_cm) and tackle each of these
> as separate problems.
>   
Use winverbs driver for connection management only and use ibal for data 
xfer ops ?

We'd still need to sort out address translation.. ?

And of course - RDS on Linux would need to connect to RDS on Windows... 
etc..

> - Sean
>
>