[openib-general] Re: uCM create connection ID

[Sean Hefty]

Libor Michalek wrote:
>>I see.  This appears to come from a difference between the event reporting 
>>model used by the kernel CM versus the usermode CM (callback versus 
>>calldown).
> 
>   Do you block the destroy on a lock while a callback for that cm_id
> is active? I wouldn't say that the difference is attributed to callback
> vs. calldown, in both cases it's a matter of serializing the destroy
> with the event.

Yes - the destroy call in the kernel blocks while there's a callback in 
progress.  After destroy returns, the CM guarantees that no additional 
callbacks will be received by the user.

The blocking destroy call was the reason for letting the user destroy the 
cm_id by returning a non-zero value from the callback.

>> Maybe there's a way to assist the user here.  Can we report a 
>>destruction event, or require a second call to indicate that an event has 
>>been processed?
> 
>   A destruction event could work, but with some limits which might make
> it impracticle. The user would have to be really carefull not to do 
> _anything_ with the object after calling destroy, and only cleanup in
> the same thread that is used to get the destroy completion event. The
> destroy completion event could be retreived and processed before the
> original destroy call returns. Also, the user would need to make sure 
> that they are getting events in a _single_ thread, since multiple event
> get threads could pose the same problem as before. 

I agree that the destroy event could occur before destroy returns, so the 
user would need to be careful there.  Arlin mentioned that there's a put 
event call that needs to be invoked after getting an event.  If so, then the 
CM can track the number of outstanding events that are in process.  It could 
then either delay the destroy call while an event is outstanding, or delay 
reporting the destroy event until all events have been processed.  This 
should handle the multi-thread issues.

It may also make sense to have the uCM serialize all events to a single 
cm_id anyway.  I ended up doing this in the kernel, which simplified the 
application's event handling.  Otherwise, the events can end up being 
processed out of order.  E.g. a REJ is reported, followed by a MRA to a REQ.

>   We could build the serialization table for the API consumer, have
> all cm_id calls and events go through a level of indirection in a
> table locked against multiple threads. This was the way we ended up
> doing it in our old code for the userCM that we used for uDAPL. I
> had left this out since it seems reasonable that not all apps would
> want/need this guarantee from the API, and that they could implement
> it themselves if they did want it... I could be wrong.

I need to spend more time looking at the uCM API/implementation to see if 
there's a way to help protect against reporting/processing events.

- Sean