[openib-general] [PATCH 5 of 5] IB/mthca: give reserved MTTs a separate cache line

[Michael S. Tsirkin]

>  > +	if (mthca_is_memfree(dev))
>  > +		dev_lim->reserved_mtts = ALIGN(1 << (field >> 4),
>  > +					       MTHCA_MTT_SEG_SIZE / sizeof(u64));
>  > +	else
>  > +		dev_lim->reserved_mtts = 1 << (field >> 4);
> 
> this still seems screwed up.  mem-free is reporting the number of
> reserved MTT entries, so I think we want (1 << log_rsvd_mtts) /
> (MTHCA_MTT_SEG_SIZE / sizeof (u64)) in that case (rounded up of
> course), rather than just aligned to something.

This issue really seems to trigger brain lockups for me.
Before I start testing, does the following look correct to you
(untested patch)?

BTW, the right thing to do I think is to get rid of MTT_SEG_SIZE for memfree
completely - as it is we are wasting MTT entries for any region that is
not a multiple of segment size. I'm just not sure I'll have the time
to fix this before 2.6.20.

Thanks,
	MST

[PATCHv4 untested 5 of 5] IB/mthca: reserved MTTs and memory alignment issues

This fixes several issues related to reserved MTTs and memory alignment.

1. MTTs are allocated in non-cache-coherent memory, so we must give
reserved MTTs their own cache line, to prevent both device and
CPU from writing into the same cache line at the same time.

2. reserved_mtts field has different meaning in Tavor and Arbel,
so we are wasting mtt entries on memfree. fix the Arbel case to match
Tavor semantics.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

---

Index: linux-2.6/drivers/infiniband/hw/mthca/mthca_main.c
===================================================================
--- linux-2.6.orig/drivers/infiniband/hw/mthca/mthca_main.c
+++ linux-2.6/drivers/infiniband/hw/mthca/mthca_main.c
@@ -464,6 +464,10 @@ static int mthca_init_icm(struct mthca_d
 		goto err_unmap_aux;
 	}
 
+	/* CPU writes to non-reserved MTTs, while HCA might DMA to reserved mtts */
+	mdev->limits.reserved_mtts = ALIGN(mdev->limits.reserved_mtts * MTHCA_MTT_SEG_SIZE,
+					   dma_get_cache_alignment()) / MTHCA_MTT_SEG_SIZE;
+
 	mdev->mr_table.mtt_table = mthca_alloc_icm_table(mdev, init_hca->mtt_base,
 							 MTHCA_MTT_SEG_SIZE,
 							 mdev->limits.num_mtt_segs,
Index: linux-2.6/drivers/infiniband/hw/mthca/mthca_cmd.c
===================================================================
--- linux-2.6.orig/drivers/infiniband/hw/mthca/mthca_cmd.c
+++ linux-2.6/drivers/infiniband/hw/mthca/mthca_cmd.c
@@ -1051,7 +1051,11 @@ int mthca_QUERY_DEV_LIM(struct mthca_dev
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_MAX_EQ_OFFSET);
 	dev_lim->max_eqs = 1 << (field & 0x7);
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_RSVD_MTT_OFFSET);
-	dev_lim->reserved_mtts = 1 << (field >> 4);
+	if (mthca_is_memfree(dev))
+		dev_lim->reserved_mtts = ALIGN((1 << (field >> 4)) * sizeof(u64),
+					       MTHCA_MTT_SEG_SIZE) / MTHCA_MTT_SEG_SIZE;
+	else
+		dev_lim->reserved_mtts = 1 << (field >> 4);
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_MAX_MRW_SZ_OFFSET);
 	dev_lim->max_mrw_sz = 1 << field;
 	MTHCA_GET(field, outbox, QUERY_DEV_LIM_RSVD_MRW_OFFSET);

-- 
MST