[openib-general] SDP_CONN_LOCK

[Libor Michalek]

On Thu, Feb 17, 2005 at 05:35:17PM +0200, Michael S. Tsirkin wrote:
> Hi, Libor!
> Could you please explain what are the SDP_CONN_LOCK
> and friends doing in SDP?
> 
> It seems they just implement exclusive access to socket,
> but if so, why is a simple semaphore or mutex not used?

  They do implement exclusive access to the socket, but they implement
exclusive access from both process and irq context, which is why a
semaphore was not used. In interrupt context SDP_CONN_LOCK_BH is used
to lock the connection, look in sdp_cq_event_handler() for it's use,
and in process context SDP_CONN_LOCK is used.

  
  The lock structure has a spinlock around a users counter, the process
context lock will increment the counter when it's using the connection,
but will not hold the spinlock for the duration. The spinlock is only
held for the duration when used in process context. In process context
the wait queue is used if the number of users is non zero, and the
wait queue is awoken when process context decrements the users counter.

  struct sdp_conn_lock {
        __u32 users;
        spinlock_t slock;
        wait_queue_head_t waitq;
  }; /* struct sdp_conn_lock */
   
   Further more if a CQ event occurs and the connection is already locked
in process context (lock.users is non-zero) the connection is marked as
having received an event and the CQ is not polled or rearmed. Instead
when the connection is unlocked using SDP_CONN_UNLOCK the CQ is drained
and rearmed. This results in a significant reduction in interrupts when
the connection is under heavy load. The fifth macro, SDP_CONN_RELOCK,
performs the equivalent of an unlock/lock in process context, which will
cause the CQ to be drained.

   The two flags SDP_CONN_F_{SEND,RECV}_CQ_PEND should not be in the
conn->flags variable, as they are currently, since that variable can be
accessed outside of the spinlock when the process context lock is held.
Instead these two flags struct sdp_conn_lock since they are only
protected when the spinlock is held.


-Libor