[openib-general] [PATCH 5 of 5] IB/mthca: give reserved MTTs a separate cache line

[Michael S. Tsirkin]

> And anyway, this is all moot I believe because the code already does:
> 
> 	for (i = 0; i < MTHCA_RES_NUM; ++i) {
> 		profile[i].type     = i;
> 		profile[i].log_num  = max(ffs(profile[i].num) - 1, 0);
> 		profile[i].size    *= profile[i].num;
> 		if (mthca_is_memfree(dev))
> 			profile[i].size = max(profile[i].size, (u64) PAGE_SIZE);
> 	}
> 
> and I'm comfortable assuming that there is no architecture where
> dma_get_cache_alignment() is bigger than PAGE_SIZE...

OK. Missed that. Good, so there's no issue.
we can drop this chunk then. So we are left with this:

[PATCHv3 5 of 5] IB/mthca: reserved MTTs issues

This fixes several issues related to reserved MTTs

1. MTTs are allocated in non-cache-coherent memory, so we must give
reserved MTTs their own cache line, to prevent both device and
CPU from writing into the same cache line at the same time.

2. reserved_mtts field has different meaning in Tavor and Arbel,
so we are wasting mtt entries on memfree. fix the Arbel case to match
Tavor semantics.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

---

Index: linux-2.6/drivers/infiniband/hw/mthca/mthca_main.c
===================================================================
--- linux-2.6.orig/drivers/infiniband/hw/mthca/mthca_main.c
+++ linux-2.6/drivers/infiniband/hw/mthca/mthca_main.c
@@ -464,6 +464,10 @@ static int mthca_init_icm(struct mthca_d
 		goto err_unmap_aux;
 	}
 
+	/* CPU writes to non-reserved MTTs, while HCA might DMA to reserved mtts */
+	mdev->limits.reserved_mtts = ALIGN(mdev->limits.reserved_mtts * MTHCA_MTT_SEG_SIZE,
+					   dma_get_cache_alignment()) / MTHCA_MTT_SEG_SIZE;
+
 	mdev->mr_table.mtt_table = mthca_alloc_icm_table(mdev, init_hca->mtt_base,
 							 MTHCA_MTT_SEG_SIZE,
 							 mdev->limits.num_mtt_segs,
Index: linux-2.6/drivers/infiniband/hw/mthca/mthca_cmd.c
===================================================================
--- linux-2.6.orig/drivers/infiniband/hw/mthca/mthca_cmd.c
+++ linux-2.6/drivers/infiniband/hw/mthca/mthca_cmd.c
@@ -1051,7 +1051,11 @@ int mthca_QUERY_DEV_LIM(struct mthca_dev
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_MAX_EQ_OFFSET);
 	dev_lim->max_eqs = 1 << (field & 0x7);
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_RSVD_MTT_OFFSET);
-	dev_lim->reserved_mtts = 1 << (field >> 4);
+	if (mthca_is_memfree(dev))
+		dev_lim->reserved_mtts = ALIGN(1 << (field >> 4),
+					       MTHCA_MTT_SEG_SIZE / sizeof(u64));
+	else
+		dev_lim->reserved_mtts = 1 << (field >> 4);
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_MAX_MRW_SZ_OFFSET);
 	dev_lim->max_mrw_sz = 1 << field;
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_RSVD_MRW_OFFSET);

-- 
MST