[ofa-general] [PATCH 2/3] RDMA/cxgb3: fix page shift calculation in build_phys_page_list()

[Roland Dreier]

 >Roland, you're quite right that the non-obvious page list is not 
 >necessarily a problem.  It causes a failure only if the virtual address 
 >that is eventually mapped to this region has an alignment with respect to 
 >large-page boundaries that is different from the alignment of the physical 
 >address.  To be concrete, the page list works as you expect if and only if
 >
 >    ((*iova_start) & ((1ULL << shift) - 1)) ==
 >        (buffer_list[0].addr & ((1ULL << shift) - 1)).

got it... I was tricking myself that the check for alignment at the
start of the function was sufficient, but it's not once we start using
a bigger value of shift.

I think the patch below should be a fix for the problem, although I've
only compile tested it.  The idea is to stop increasing shift once it
reaches a bit position where the first buffer and the iova differ.
What do you think?  If this works for you, I will merge it for 2.6.25.

diff --git a/drivers/infiniband/hw/mthca/mthca_provider.c b/drivers/infiniband/hw/mthca/mthca_provider.c
index 6bcde1c..1a15129 100644
--- a/drivers/infiniband/hw/mthca/mthca_provider.c
+++ b/drivers/infiniband/hw/mthca/mthca_provider.c
@@ -948,7 +948,9 @@ static struct ib_mr *mthca_reg_phys_mr(struct ib_pd       *pd,
 		return ERR_PTR(-EINVAL);
 
 	/* Find largest page shift we can use to cover buffers */
-	for (shift = PAGE_SHIFT; shift < 31; ++shift)
+	for (shift = PAGE_SHIFT; shift < 31; ++shift) {
+		if ((buffer_list[0].addr ^ *iova_start) & (1ULL << shift))
+			break;
 		if (num_phys_buf > 1) {
 			if ((1ULL << shift) & mask)
 				break;
@@ -958,6 +960,7 @@ static struct ib_mr *mthca_reg_phys_mr(struct ib_pd       *pd,
 			    (buffer_list[0].addr & ((1ULL << shift) - 1)))
 				break;
 		}
+	}
 
 	buffer_list[0].size += buffer_list[0].addr & ((1ULL << shift) - 1);
 	buffer_list[0].addr &= ~0ull << shift;