[openib-general] mthca FMR correctness (and memory windows)

[Talpey, Thomas]

At 01:01 AM 3/21/2006, Dror Goldenberg wrote:
>Not sure we managed to convince you anything about FMRs. Anyway,

On the contrary, I feel I know them much better. :-) I'm certainly
more aware of the behavior of the fmr pool code, which is not
appropriate for storage ULPs, in my opinion.

>I would suggest, even just for the sake of performance evaluation to
>try the following FMR approaches:
>0a - this is the allegedly fastest: using FMR to consolidate list of
>pages
>2a - with fmr map/unmap - same behavior as MWs (this is what you called
>4)
>3a - with fmr pool - will be like async unbind.

Yes, I am considering these. You're reversing my numbering, but I can
say that your 0a is definitely desirable, and 2a is what I'm attempting to
implement now.

>I wouldn't be surprised if you end up finding 0a a win-win for both the
>client
>and the server. If you end up finding differently, then that may also be
>interesting.
>BTW, iSER only works this way, the RFC does not allow passing a 
>chunk list as far as I know...

Yes, iSER follows the SCSI transfer mode which places a single segment
on the wire for each operation. RPC/RDMA was designed rather differently.
For one thing, NFS is not a block-oriented protocol. This means it is more
flexible w.r.t. data segmentation. Also, NFS has a much broader range of
message types, with metadata payload. These lead to requirements for a
more flexible wire structure.

I am hopeful that NFS/RDMA will lend itself well to cluster computing, due
to its good sharing semantics, transparent file API, and low overhead from
use of the RDMA fabric. The one thing I don't want to build in is some
kind of compromise on security or data integrity. No performance gain is
worth that.

Tom.