[openib-general] qp lock in mthca_poll_cq

[Roland Dreier]

There are two separate uses of the QP during CQ poll, which is why
refcount is an atomic_t and we also take the spinlock.

First, the refcount is used to make sure that destroy_qp does not
get rid of the QP struct while the QP is being accessed to handle
poll_cq.  It is an atomic_t because it may be accessed without the QP
lock being held (eg when an async event is received for the QP).

Second, the QP's lock is taken during CQE processing because other
non-atomic parts of the QP struct such as the number of WQEs
outstanding _are_ modified and need to be protected against concurrent
access from the send/receive post routine.

It might be possible to avoid taking the QP lock in poll_cq by making
the current WQE count an atomic_t, but I'm not sure if that's a win
because it means that send/receive posting would have to use atomic
accesses as well (and I don't think you can make posting WQEs lock-free).

Some of my reasoning is in the comment near the bottom of
mthca_provider.h too.

 - Roland