[ofa-general] [PATCH v2] IB/mlx4: Add send with invalidate support

[Or Gerlitz]

Eli Cohen wrote:
> Add send with invalidate support to mlx4.
> Signed-off-by: Eli Cohen <eli at mellanox.co.il>
Hi Eli,

Thinking about it a little, I don't see how this send-with-invalidate 
support is applicable for ULPs such as SRP, iSER and RDS who use the 
Mellanox FMRs.

This is because to invalidate an rkey they call fmr_unmap and what the 
mlx4 (similarly for mthca) driver does at mlx4_ib_unmap_fmr() is call 
mlx4_fmr_unmap() for each fmr and then issue SYNC_TPT command. 

Even if doing send-with-inv would save the ULP the indirect call to 
mlx4_fmr_unmap() (which does almost nothing by itself), if it doesn't 
cause the HW/FW to issue SYNC_TPT, it can not replace the call to 
ib_unmap_fmr in the side that generated this rkey. And if it does cause 
SYNC_TPT, the effect of amortizing the cost of this heavy command 
through un-mapping on many fmrs at once is lost, correct?

Or
> mlx4_ib_unmap_fmr calls mlx4_fmr_unmap for each fmr and then issues SYNC_TPT command
>
> void mlx4_fmr_unmap(struct mlx4_dev *dev, struct mlx4_fmr *fmr,
>                     u32 *lkey, u32 *rkey)
> {
>         if (!fmr->maps)
>                 return;
>
>         fmr->maps = 0;
>
>         *(u8 *) fmr->mpt = MLX4_MPT_STATUS_SW;
> }
>