[ofa-general] rping / librdmacm deadlock question

[Roland Dreier]

 > Our driver (as do all drivers I've seen) performs an
 > atomic_dec(&qp->refcount) and wait_event(&qp->refcount) in 'destroy_qp()'.
 > Because the rdma_cm device hasn't been closed (i.e., ucma_close() hasn't yet
 > been called), a cm_id still has an active reference to the qp, and the
 > wait_event() will end up 'wait'ing.

In the other drivers I know well (basically mthca and mlx4, since I
wrote them), the qp->refcount being waited for is an internal driver
refcount, and is used to make sure that the destroy QP operation waits
until any active interrupt handlers are done with the QP.  So I think
the problem is that you are letting a cm_id bump the QP's reference
count somehow.

 > Perhaps my device driver should do additional work in ib_destroy_qp() that
 > will trigger the destruction of the cm_id... [but that doesn't seem
 > consistent with other drivers I've seen.]

That doesn't make sense.  I think it's OK if upper layers are left
with a stale pointer to your QP -- let them worry about it.  Maybe
it's an iWARP thing that I don't really understand (I'm much more
familiar with the IB driver interface) but I don't think that the
cxgb3 driver runs into this issue.

 > Perhaps the application (i.e., librdmacm) should make sure the "rdma_cm"
 > device is closed before calling ibv_device_close()?

No, because then some other (possibly malicious) app could still cause
the deadlock and potentially create a bunch of unkillable processes.

 - R.