[openib-general][patch review] srp: fmr implementation,

[Vu Pham]

>     Vu> I don't have a good test to measure performance; however, I
>     Vu> ran multiple dd streams on raw devices (no filesystem or
>     Vu> buffer cache involved) to measure the impact of using
>     Vu> FMRs. With Mellanox srp target referenced implementation SW +
>     Vu> 14 SATA drives + PCI-E lion cub, I hit 850 MB/s with FMRs and
>     Vu> 710 MB/s without it.
> 
> Hmm, I'm not sure whether dd is a good test.  I'd be more interested
> to know if typical filesystem I/O generates requests that are
> mergeable via FMRs.
> 

Probably I will run some IOZONE and compare


>     Vu> Please locally apply the patch, measure the performance impact
>     Vu> and post any numbers if possible with your setup
> 
> Yes, I need to do that, and also add some instrumentation to see how
> often FMRs are used on different workloads.
> 

That's a good idea.
Currently the patch only tries to merge all sg elems per cmd 
to single FMR if possible; otherwise, it falls back to dma 
mr. We could improve the implementation to merge all sg 
elems per cmd to 2-3 FMRs or mix dma mr with FMR

Vu