[ofw] ibal library callback model

[Fab Tillier]

> I'm trying to understand if a user can get simultaneous callbacks for
> the same CM connection.

No, the CM callbacks are serialized by CEP.  In ual_cm_cep.c, in the function cm_cb (line 1423), you can see the CEP callback being invoked before the next UAL_CEP_GET_EVENT IOCTL is issued.

> For example, I see the following call paths in ual_mgr.c:
>
> __cb_thread_routine() -> cm_cb() -> __cm_handler() ->
> __proc_conn() -> __proc_dreq()
>
> I end up with two threads in this same code path.  I wanted to verify
> that the same connection would not get simultaneous callbacks, and I
> wanted to confirm that blocking in the callback could lead to a
> deadlock, depending on what was being waited on.  In this case, blocking
> in a CM callback affects callbacks for CQs.

Two threads in that code path would have to be for different CEPs (unless there's a bug), the same connection will not get simultaneous callbacks.  Originally the kernel code would push events to UAL, and that required more serialization in user-mode.  Internally, the CEP manager works very much like WinVerbs/ND with an async IOCTL outstanding until the next CEP event.  The kernel does not push events to user-mode, user-mode must explicitly request events.  Since the event is just a notification event, the al_cm_qp.c code path ends up polling the CEP for the actual events.

If you end up blocking all the callback threads, you will potentially deadlock. This is another reason I dislike the callback model and prefer the 'thread less' model of ND/WinVerbs, as all blocking/deadlocking logic is in the client's domain and requires no special knowledge of how the provider is implemented.

-Fab