[openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Roland Dreier]

    Shirley> I don't think two atomic operation is that expensive
    Shirley> compare to reaping AHs in process context according to
    Shirley> the test results and profiling data. Or we can use RCU
    Shirley> instead.

What is the cost of reaping AHs?  Under most workloads I would expect AHs
essentially never get freed.

Are you saying that you see __ipoib_reap_ah() show up in your profiles?
If so then we should fix the underlying cause of that rather than
slowing down the ipoib fast path.

BTW, since it is allowed to call ib_destroy_ah() with a lock held (cf
Documentation/infiniband/core_locking.txt) then I think the patch
below is a reasonable cleanup:

--- infiniband/ulp/ipoib/ipoib_ib.c	(revision 7485)
+++ infiniband/ulp/ipoib/ipoib_ib.c	(working copy)
@@ -380,15 +380,10 @@ static void __ipoib_reap_ah(struct net_d
 	list_for_each_entry_safe(ah, tah, &priv->dead_ahs, list)
 		if ((int) priv->tx_tail - (int) ah->last_send >= 0) {
 			list_del(&ah->list);
-			list_add_tail(&ah->list, &remove_list);
+			ib_destroy_ah(ah->ah);
+			kfree(ah);
 		}
 	spin_unlock_irq(&priv->lock);
-
-	list_for_each_entry_safe(ah, tah, &remove_list, list) {
-		ipoib_dbg(priv, "Reaping ah %p\n", ah->ah);
-		ib_destroy_ah(ah->ah);
-		kfree(ah);
-	}
 }
 
 void ipoib_reap_ah(void *dev_ptr)