[ofa-general] [PATCH RFC] sharing userspace IB objects

[Or Gerlitz]

Michael S. Tsirkin wrote:

>> So, since the need seems to be there, I started thinking about how this could be done.
>> Basically, we could create shared memory objects (shm_open) and use these
>> for all hardware-accessible registers, as well as necessary control (head/tail pointers,
>> spinlocks used for protection, etc).
>>
>> If we do this, we can use unix domain sockets for everything,
>> a client just mmaps the fd that it got. Does this make sense?
> 
> OK, here's a draft showing how an API to do this could look like.
> 
> Basically the idea is that we'd ask low-level drivers to provide an
> (optional) API to
> 1. allocate context and all its objects inside a shared memory object
> 2. pack and unpack objects from/to unix domain socket messages
> 
> So to share a QP, the server would
> A. open shared context, create pd, cq, qp
> B. listen on unix domain socket
> C. pack the context, pd, cq, qp
> D. send them to clients that connect

> The client would
> A. create unix domain socket
> B. connect to server
> C. get message from server
> D. unpack context, pd, cq, qp

One problem here (which annoys for long time...) is that typically the 
active side of a connection is the one that sends the first packet and 
hence you must post receives to the QP --before-- accepting the 
connection request.

So, if both sides use a shared-context, they would need to implement a 
synchronization protocol (that is don't deliver established event to the 
active before the passive accepted).

And, if the active side does not use shared context where the passive 
side does use shared context, you need either the shared context to 
allocate/post receives from shared-memory or rely on RNR NAKs, what do 
you think?

Also, what was your thinking on registering the QP/CQ memory? is the 
plan to implement a verb for registering shared-memory as was in the 
VAPI stack, or you want to register this memory as "just" virtual?

Or.