[openib-general][patch review] srp: fmr implementation,

[Vu Pham]

>>
>>>BTW, does Mellanox (or anyone else) have any numbers showing that
>>>using FMRs makes any difference in performance on a semi-realistic benchmark?
>>>
>>
>>I'm using xdd to test the performance 
>>www.ioperformance.com/products.htm
>>
>>The target is Mellanox srp target reference implemenation 
>>with 14 SATA spindles
>>
>>I can get ~780 MB/s max without FMRs and ~920 MB/s with FMRs 
>>(using 256 KB sequential read direct IO request)
> 
> 
> Wow, that's awesome Vu! 
> 
> So what's the consensus on the reason for the improvement? 
> - Fewer WR to send the same amount of data because the memory is
> virtually contiguous?

Fewer wqe, fewer interrupts & context switches, big I/O 
request directly sent to back-end storage/spindles (all are 
applied for target)

For write command (RDMA read), it help to utilize the max 
outstanding rdma read operations (more data transfer per 
outstanding rdma read operations)

> - Fewer PDU due to larger writes and better packing?

This is not much.

> - Something else?
> 
> Are these huge reads, ... or even better ... what are the parameters to
> xdd?


-op read -reqsize 128 -block 2048 -dio -mbytes 8192 
-targetdir /dev/ -targets 14 sdb sdc sdd ....


> 
> Did you quantify the FMR registration cost per MB of registered memory
> space? This would be a very good number to have for figuring out where
> the sweet spot is...

I don't have this number of FMR registration cost per MB.

Dror, do we have any number on how many cycles cost per FMR 
registration?

On the side note the FMR registration hit rate is ~100% with 
direct I/O

Vu