[ofa-general] [PATCH v2] IB/mlx4: Add send with invalidate support

[Eli Cohen]

On Mon, 2008-05-12 at 08:44 +0300, Or Gerlitz wrote:

> Thinking about it a little, I don't see how this send-with-invalidate 
> support is applicable for ULPs such as SRP, iSER and RDS who use the 
> Mellanox FMRs.
> 
> This is because to invalidate an rkey they call fmr_unmap and what the 
> mlx4 (similarly for mthca) driver does at mlx4_ib_unmap_fmr() is call 
> mlx4_fmr_unmap() for each fmr and then issue SYNC_TPT command. 
> 
> Even if doing send-with-inv would save the ULP the indirect call to 
> mlx4_fmr_unmap() (which does almost nothing by itself), if it doesn't 
> cause the HW/FW to issue SYNC_TPT, it can not replace the call to 
> ib_unmap_fmr in the side that generated this rkey. And if it does cause 
> SYNC_TPT, the effect of amortizing the cost of this heavy command 
> through un-mapping on many fmrs at once is lost, correct?
The outcome of send with invalidate involves an implicit "sync_tpt" like
operation although it syncs the caches to invalidate only the specific
memory key (as opposed to sync tpt command which has a more global
nature).
But I think that the idea is not to save the overhead of sync tpt
commands but to provide security. Perhaps someone from RDS can add more
on that.