[openib-general][patch review] srp: fmr implementation,

[Grant Grundler]

On Tue, Mar 14, 2006 at 07:43:41AM -0800, Roland Dreier wrote:
>     Vu> I don't have a good test to measure performance; however, I
>     Vu> ran multiple dd streams on raw devices (no filesystem or
>     Vu> buffer cache involved) to measure the impact of using
>     Vu> FMRs. With Mellanox srp target referenced implementation SW +
>     Vu> 14 SATA drives + PCI-E lion cub, I hit 850 MB/s with FMRs and
>     Vu> 710 MB/s without it.
> 
> Hmm, I'm not sure whether dd is a good test.

dd is not since dd doesn't validate the result of read or write.
Try "spew".

For "raw" performance, I prefer "sgp_dd" for maximising throughput
to "normal" SCSI disks:
grundler at gsyprf11:~$ dpkg -S sgp_dd
sg3-utils: /usr/bin/sgp_dd


> I'd be more interested to know if typical filesystem I/O generates
> requests that are mergeable via FMRs.

How would we know if FMRs are merging transactions?
(more on this below)

>     Vu> Do you have a good test to measure performance?
> 
> No, I don't.  That's why I ask.

spew gives some perf metrics.
Also see iozone, bonnie++, and diskbench.

> Yes, I need to do that, and also add some instrumentation to see how
> often FMRs are used on different workloads.

ah ok. I was hoping HW (HCA) could tell us.

Keeping stats can cost substantial performance because of additional
cacheline activity (read/modify/write operations).  I ripped out the
"interesting" (to me only) stats in my IOMMU code for this reason.

grant