[openib-general] [PATCHv5 5 of 5] IB/mthca: reserved MTTs and memory alignment issues

[Michael S. Tsirkin]

This fixes several issues related to reserved MTTs and memory alignment.

1. MTTs are allocated in non-cache-coherent memory, so we must give
reserved MTTs their own cache line, to prevent both device and
CPU from writing into the same cache line at the same time.

2. reserved_mtts field has different meaning in Tavor and Arbel,
so we are wasting mtt entries on memfree. fix the Arbel case to match
Tavor semantics.

2. We are allocating separate ICM tables for each resource, so each must
be aligned at ICM page boundary (existing code happened to work fine so
far as profile was hard-coded with values that are all high powers of
2, but with new module option code this might not be the case anymore).
And since we access some of them from CPU and some from hardware, we
really want to give different tables separate dma cache lines too.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

---

This version works for me.

Two side notes
1. This replaces the v1 of the patch in the series
(which was titled [PATCH 5 of 5] IB/mthca: give reserved MTTs a separate
cache line) and addresses the two issues raised.
Roland, please let me know if you want this handled differently
(e.g. should I repost all patches in the series, or split this one to
3 one-liners?).

2. memfree does not actually have a notion of mtt segments.
So it seems we could save some mtt entries by making mtt segment
size device-dependent. This is a win if there are lots of small regions,
this optimization needs to be looked at separately.

Index: linux-2.6/drivers/infiniband/hw/mthca/mthca_main.c
===================================================================
--- linux-2.6.orig/drivers/infiniband/hw/mthca/mthca_main.c
+++ linux-2.6/drivers/infiniband/hw/mthca/mthca_main.c
@@ -464,6 +464,10 @@ static int mthca_init_icm(struct mthca_d
 		goto err_unmap_aux;
 	}
 
+	/* CPU writes to non-reserved MTTs, while HCA might DMA to reserved mtts */
+	mdev->limits.reserved_mtts = ALIGN(mdev->limits.reserved_mtts * MTHCA_MTT_SEG_SIZE,
+					   dma_get_cache_alignment()) / MTHCA_MTT_SEG_SIZE;
+
 	mdev->mr_table.mtt_table = mthca_alloc_icm_table(mdev, init_hca->mtt_base,
 							 MTHCA_MTT_SEG_SIZE,
 							 mdev->limits.num_mtt_segs,
Index: linux-2.6/drivers/infiniband/hw/mthca/mthca_cmd.c
===================================================================
--- linux-2.6.orig/drivers/infiniband/hw/mthca/mthca_cmd.c
+++ linux-2.6/drivers/infiniband/hw/mthca/mthca_cmd.c
@@ -1051,7 +1051,11 @@ int mthca_QUERY_DEV_LIM(struct mthca_dev
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_MAX_EQ_OFFSET);
 	dev_lim->max_eqs = 1 << (field & 0x7);
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_RSVD_MTT_OFFSET);
-	dev_lim->reserved_mtts = 1 << (field >> 4);
+	if (mthca_is_memfree(dev))
+		dev_lim->reserved_mtts = ALIGN(1 << (field >> 4),
+					       MTHCA_MTT_SEG_SIZE / sizeof(u64));
+	else
+		dev_lim->reserved_mtts = 1 << (field >> 4);
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_MAX_MRW_SZ_OFFSET);
 	dev_lim->max_mrw_sz = 1 << field;
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_RSVD_MRW_OFFSET);
Index: linux-2.6/drivers/infiniband/hw/mthca/mthca_profile.c
===================================================================
--- linux-2.6.orig/drivers/infiniband/hw/mthca/mthca_profile.c
+++ linux-2.6/drivers/infiniband/hw/mthca/mthca_profile.c
@@ -39,6 +39,7 @@
 #include <linux/slab.h>
 
 #include "mthca_profile.h"
+#include "mthca_memfree.h"
 
 enum {
 	MTHCA_RES_QP,
@@ -147,6 +148,10 @@ u64 mthca_make_profile(struct mthca_dev 
 
 	for (i = 0; i < MTHCA_RES_NUM; ++i) {
 		if (profile[i].size) {
+			if (mthca_is_memfree(dev))
+				total_size = ALIGN(total_size, max(MTHCA_ICM_PAGE_SIZE,
+								   dma_get_cache_alignment()));
+
 			profile[i].start = mem_base + total_size;
 			total_size      += profile[i].size;
 		}


-- 
MST