[ofa-general] iSER data corruption issues

[Pete Wyckoff]

Thomas.Talpey at netapp.com wrote on Thu, 04 Oct 2007 07:55 -0400:
> This is an aside, but in my experience the FMR is actually a win even if
> it's invalidated after each use. In testing with NFS/RDMA, I believe that
> direct FMR manipulation via ib_map_phys_mr()/ib_unmap_fmr() was worth
> somewhere on the order of 35% over straight ib_reg_phys_mr()/ib_dereg_mr().
> I can only assume this was because the TPT-entry setup (ib_alloc_fmr())
> is avoided on a per-I/O basis.
> 
> As for the pools not invalidating the R_key/handle. Speaking as a storage
> provider, we take data integrity darn seriously. It's my opinion that a
> dynamic registration scheme that doesn't include per-I/O protection is
> pretty much not the point of dynamic registration. In many environments
> however, the performance tradeoff is important - this is why I prefer an
> all-physical scheme to FMRs, even though it requires additional RDMA ops
> to handle the resulting extra scatter/gather.

Ack.  Unfortunately in the iSER case, we are limited to a single
virtual address per command.  Page size fragmentation may destroy
performance, even with heavy pipelining.

		-- Pete

> Additionally, FMRs don't provide byte-range protection granularity,
> and they're not supported by iWARP hardware (plus they're buggy
> as heck on early Tavors, etc). So I didn't make them a default.