[ofa-general] mthca max_sge value... ugh.

[Jack Morgenstein]

On Saturday 17 May 2008 02:12, Roland Dreier wrote:
> 
> if we can't use the "WQE shrinking" feature (because of selective
> signaling in the NFS/RDMA case), and we want to use 32 sge entries, then
> the WQE size 's' will end up a little more than 512 bytes, and the
> wqe_shift will end up as 10.

> But since the max_sq_desc_sz is 1008, we 
> return -EINVAL, when it is really fine to have a wqe_shift of 10 as long
> as we don't use more than 1008 bytes per descriptor (I think).
Correct. 

...

> @@ -395,7 +396,8 @@ static int set_kernel_sq_size(struct mlx4_ib_dev *dev, struct ib_qp_cap *cap,
>  		++qp->sq.wqe_shift;
>  	}
>  
> -	qp->sq.max_gs = ((qp->sq_max_wqes_per_wr << qp->sq.wqe_shift) -
> +	qp->sq.max_gs = (min(dev->dev->caps.max_sq_desc_sz,
> +			     (qp->sq_max_wqes_per_wr << qp->sq.wqe_shift)) -
>  			 send_wqe_overhead(type, qp->flags)) /
>  		sizeof (struct mlx4_wqe_data_seg);

In this case, sq.max_gs ( = (1008 - wqe overhead) / 16) will be larger than the
"max sge" value returned by ib_query_device, (max_sge returned by ib_query_device is 32).
I'm not crazy about this inconsistency.  Please note also that the IB Spec does not
differentiate between Send max_sge, and Receive max_sge, so we're reduced to enforcing
the minimum of the two values.

The general approach taken in the driver is to enforce the smallest of the sge values,
to avoid dealing with the individual qp type maxima.

- Jack