[openib-general] [PATCH] mthca: eq doorbell coalescing + prevent even queque overrun

[Michael S. Tsirkin]

I am seeing EQ overruns in SDP stress tests: if CQ completion
handler arms a CQ, this could generate more EQEs, so that
EQ will never get empty and consumer index will never get updated.

This is similiar to what we have with command interface:
		/*
		 * cmd_event() may add more commands.
		 * The card will think the queue has overflowed if
		 * we don't tell it we've been processing events.
		 */
However, for completion events, we *dont* want to update the consumer index on
each event. So, perform EQ doorbell coalescing: allocate an EQ with some spare
EQEs, and update once we run out of them.

The value 0x80 was selected to avoid any performance impact.

---

Fix EQ overrun for completion events: increase EQ size by 0x80,
and tell the card that we have been processing events, at least each 0x80 EQEs.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.14.3/drivers/infiniband/hw/mthca/mthca_eq.c
===================================================================
--- linux-2.6.14.3.orig/drivers/infiniband/hw/mthca/mthca_eq.c	2006-01-05 18:11:04.000000000 +0200
+++ linux-2.6.14.3/drivers/infiniband/hw/mthca/mthca_eq.c	2006-01-08 09:55:28.000000000 +0200
@@ -45,6 +45,7 @@
 enum {
 	MTHCA_NUM_ASYNC_EQE = 0x80,
 	MTHCA_NUM_CMD_EQE   = 0x80,
+	MTHCA_NUM_SPARE_EQE = 0x80,
 	MTHCA_EQ_ENTRY_SIZE = 0x20
 };
 
@@ -277,11 +278,10 @@ static int mthca_eq_int(struct mthca_dev
 {
 	struct mthca_eqe *eqe;
 	int disarm_cqn;
-	int  eqes_found = 0;
+	int eqes_found = 0;
+	int set_ci = 0;
 
 	while ((eqe = next_eqe_sw(eq))) {
-		int set_ci = 0;
-
 		/*
 		 * Make sure we read EQ entry contents after we've
 		 * checked the ownership bit.
@@ -345,12 +345,6 @@ static int mthca_eq_int(struct mthca_dev
 					be16_to_cpu(eqe->event.cmd.token),
 					eqe->event.cmd.status,
 					be64_to_cpu(eqe->event.cmd.out_param));
-			/*
-			 * cmd_event() may add more commands.
-			 * The card will think the queue has overflowed if
-			 * we don't tell it we've been processing events.
-			 */
-			set_ci = 1;
 			break;
 
 		case MTHCA_EVENT_TYPE_PORT_CHANGE:
@@ -385,8 +379,14 @@ static int mthca_eq_int(struct mthca_dev
 		set_eqe_hw(eqe);
 		++eq->cons_index;
 		eqes_found = 1;
+		++set_ci;
 
-		if (unlikely(set_ci)) {
+		/*
+		 * The card will think the queue has overflowed if
+		 * we don't tell it we've been processing events,
+		 * now and then.
+		 */
+		if (unlikely(set_ci >= MTHCA_NUM_SPARE_EQE)) {
 			/*
 			 * Conditional on hca_type is OK here because
 			 * this is a rare case, not the fast path.
@@ -862,19 +862,19 @@ int __devinit mthca_init_eq_table(struct
 	intr = (dev->mthca_flags & MTHCA_FLAG_MSI) ?
 		128 : dev->eq_table.inta_pin;
 
-	err = mthca_create_eq(dev, dev->limits.num_cqs,
+	err = mthca_create_eq(dev, dev->limits.num_cqs + MTHCA_NUM_SPARE_EQE,
 			      (dev->mthca_flags & MTHCA_FLAG_MSI_X) ? 128 : intr,
 			      &dev->eq_table.eq[MTHCA_EQ_COMP]);
 	if (err)
 		goto err_out_unmap;
 
-	err = mthca_create_eq(dev, MTHCA_NUM_ASYNC_EQE,
+	err = mthca_create_eq(dev, MTHCA_NUM_ASYNC_EQE + MTHCA_NUM_SPARE_EQE,
 			      (dev->mthca_flags & MTHCA_FLAG_MSI_X) ? 129 : intr,
 			      &dev->eq_table.eq[MTHCA_EQ_ASYNC]);
 	if (err)
 		goto err_out_comp;
 
-	err = mthca_create_eq(dev, MTHCA_NUM_CMD_EQE,
+	err = mthca_create_eq(dev, MTHCA_NUM_CMD_EQE + MTHCA_NUM_SPARE_EQE,
 			      (dev->mthca_flags & MTHCA_FLAG_MSI_X) ? 130 : intr,
 			      &dev->eq_table.eq[MTHCA_EQ_CMD]);
 	if (err)

-- 
MST