[openib-general] Re: RDMA Generic Connection Management

[James Lentini]

On Fri, 2 Sep 2005, Roland Dreier wrote:

>     Roland> Yes, but what is the generic way?
> 
>     James> The generic way would be to handle this in a common
>     James> layer. For the IB verbs + RDMA connection API to be as easy
>     James> to use as the sockets API, then it needs to make this issue
>     James> transparent.
> 
> I don't think the kernel design philosophy is to hide these sorts of
> object lifetime issues from consumers.  

I disagree. The kernel Sockets API makes these issues transparent. 

The interactions between the PPPoE driver and the Sockets layer are a 
nice example. The PPPoE driver registers a callback for network device 
events by calling register_netdevice_notifier(), which is similar to 
our ib_register_client() function. When a hotplug event occurs, the 
ppoe_flush_dev() function is called. This function cleans up all 
sockets on the device that is being removed. For each socket, this 
function obtains a lock on the socket via lock_sock(). The 
pppoe_sendmsg() function (which is invoked by net/socket.c's 
kernel_sendmsg() function for PPPoE sockets) also obtains this lock 
via lock_sock(). Therefore the socket lock ensures that a consumer can 
send a message without worrying about hotplug events.

> And my real question is how it's even possible to handle this 
> efficiently in a generic layer.
>
> If you want consumers to be able to ignore hotplug, then the generic
> layer needs to handle device removal even in the middle of fast path
> work request posting operations.  And I don't see how to do that
> without changing to reference counted handles (from the current scheme
> of directly using pointers).  And that's going to have a serious
> performance impact that I don't think is worth it.

I agree that if hotplug is hidden inside the common RDMA layer, 
additional locking will be necessary. 

I think the performance impact will vary depending on the ULP. 

Some ULPs (SRP) connect into layers (SCSI) that are designed for 
hotplug. For this class of ULPs, hiding hotplug events inside the 
common RDMA layer is unnecessary. 

There is another class of ULPs (NFS RPC) that currently use the 
Sockets API but want to start using RDMA. For these ULPs to use the 
common RDMA layer they need additional locking to deal with hotplug. 
The locking could be done in a common layer or in each ULP. I expect 
both these options to have the same performance. However, 
consolidating the locking into a common layer will benefit 
correctness, code reuse, and ease the task of moving ULPs from the 
sockets API to RDMA.