[ofa-general] Re: [PATCH 2 of 3 for-2.6.24] mlx4: always fill MTTs from CPU

[Roland Dreier]

 > +	num_mtt_res_bytes = ((dev->caps.reserved_mtts *
 > +			      (dev->caps.mtt_entry_sz / MLX4_MTT_ENTRY_PER_SEG)
 > +			     + L1_CACHE_BYTES - 1) /
 > +			     L1_CACHE_BYTES) * L1_CACHE_BYTES;
 >  	err = mlx4_init_icm_table(dev, &priv->mr_table.mtt_table,
 >  				  init_hca->mtt_base,
 >  				  dev->caps.mtt_entry_sz,
 >  				  dev->caps.num_mtt_segs,
 > -				  dev->caps.reserved_mtts, 1, 0);
 > +				  num_mtt_res_bytes / dev->caps.mtt_entry_sz,
 > +				  1, 0);

This is a little off I think because it may be that mtt_entry_sz might
be bigger than L1_CACHE_BYTES so the number of reserved objects might
not be big enough.  Also the current meaning of reserved_mtts is
really wrong (it leads to the driver reserving too much stuff) -- we
should convert it to segments, like this (I'll put this in my queue
before this patch and fix up the expression above):

diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c
index 07c2847..ed7e8d7 100644
--- a/drivers/net/mlx4/main.c
+++ b/drivers/net/mlx4/main.c
@@ -149,7 +149,8 @@ static int __devinit mlx4_dev_cap(struct mlx4_dev *dev, struct mlx4_dev_cap *dev
 	dev->caps.max_cqes	     = dev_cap->max_cq_sz - 1;
 	dev->caps.reserved_cqs	     = dev_cap->reserved_cqs;
 	dev->caps.reserved_eqs	     = dev_cap->reserved_eqs;
-	dev->caps.reserved_mtts	     = dev_cap->reserved_mtts;
+	dev->caps.reserved_mtts	     = DIV_ROUND_UP(dev_cap->reserved_mtts,
+						    MLX4_MTT_ENTRY_PER_SEG);
 	dev->caps.reserved_mrws	     = dev_cap->reserved_mrws;
 	dev->caps.reserved_uars	     = dev_cap->reserved_uars;
 	dev->caps.reserved_pds	     = dev_cap->reserved_pds;