[ofa-general] iSER data corruption issues

[Talpey, Thomas]

At 11:09 PM 10/3/2007, Roland Dreier wrote:
>...  It just keeps a list of FMRs that are available to remap, and
>batches up the unregistration.  It is true that an R_Key may remain
>valid after an FMR is unmapped, but that's the whole point of FMRs: if
>you don't batch up the real flushing to amortize the cost, they're no
>better than regular MRs really.

This is an aside, but in my experience the FMR is actually a win even if
it's invalidated after each use. In testing with NFS/RDMA, I believe that
direct FMR manipulation via ib_map_phys_mr()/ib_unmap_fmr() was worth
somewhere on the order of 35% over straight ib_reg_phys_mr()/ib_dereg_mr().
I can only assume this was because the TPT-entry setup (ib_alloc_fmr())
is avoided on a per-I/O basis.

As for the pools not invalidating the R_key/handle. Speaking as a storage
provider, we take data integrity darn seriously. It's my opinion that a
dynamic registration scheme that doesn't include per-I/O protection is
pretty much not the point of dynamic registration. In many environments
however, the performance tradeoff is important - this is why I prefer an
all-physical scheme to FMRs, even though it requires additional RDMA ops
to handle the resulting extra scatter/gather.

Additionally, FMRs don't provide byte-range protection granularity,
and they're not supported by iWARP hardware (plus they're buggy
as heck on early Tavors, etc). So I didn't make them a default.

Tom.