[ofa-general] [PATCH] mlx4: fix post send of local invalidate and fast registration packets.

[Tom Talpey]

On 6/4/2009 10:40 AM, Jack Morgenstein wrote:
> The ConnectX PRM states that the "SO" bit must be set when posting Fast Reg
> and Local Invalidate send packets. When this bit is set, the current WQE will
> be executed only after all previous WQEs on this QP have been executed.
> (If the bit is not set, Fast Reg and Local Invalidate WQEs may begin execution
> too early).
>
> Signed-off-by: Jack Morgenstein<jackm at dev.mellanox.co.il>
>
> ---
> Roland,
> Please try to get this patch into 2.6.30 -- it is an important fix for nfsrdma.

I wasn't aware of this - tell me more what is happening in NFS/RDMA?

There is a problem in the NFS client where it isn't setting the
LOCAL_WRITE attribute in the FRMR code path, which causes a problem
on ConnectX if you don't use the workaround to modify the client's
memreg sysctl. That bug is fixe and currently queued in Linus' tree
for 2.6.30. If you want, I'll dig up the git change.

Tom.

> Thanks!
>
> diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c
> index 20724ae..c4a0264 100644
> --- a/drivers/infiniband/hw/mlx4/qp.c
> +++ b/drivers/infiniband/hw/mlx4/qp.c
> @@ -1585,12 +1585,16 @@ int mlx4_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
>   				break;
>
>   			case IB_WR_LOCAL_INV:
> +				ctrl->srcrb_flags |=
> +					cpu_to_be32(MLX4_WQE_CTRL_STRONG_ORDER);
>   				set_local_inv_seg(wqe, wr->ex.invalidate_rkey);
>   				wqe  += sizeof (struct mlx4_wqe_local_inval_seg);
>   				size += sizeof (struct mlx4_wqe_local_inval_seg) / 16;
>   				break;
>
>   			case IB_WR_FAST_REG_MR:
> +				ctrl->srcrb_flags |=
> +					cpu_to_be32(MLX4_WQE_CTRL_STRONG_ORDER);
>   				set_fmr_seg(wqe, wr);
>   				wqe  += sizeof (struct mlx4_wqe_fmr_seg);
>   				size += sizeof (struct mlx4_wqe_fmr_seg) / 16;
> diff --git a/include/linux/mlx4/qp.h b/include/linux/mlx4/qp.h
> index bf8f119..9f29d86 100644
> --- a/include/linux/mlx4/qp.h
> +++ b/include/linux/mlx4/qp.h
> @@ -165,6 +165,7 @@ enum {
>   	MLX4_WQE_CTRL_IP_CSUM		= 1<<  4,
>   	MLX4_WQE_CTRL_TCP_UDP_CSUM	= 1<<  5,
>   	MLX4_WQE_CTRL_INS_VLAN		= 1<<  6,
> +	MLX4_WQE_CTRL_STRONG_ORDER	= 1<<  7,
>   };
>
>   struct mlx4_wqe_ctrl_seg {
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>