[openib-general] mthca FMR correctness (and memory windows)

[Dror Goldenberg]

> From: Talpey, Thomas [mailto:Thomas.Talpey at netapp.com] 
> Sent: Monday, March 20, 2006 11:50 PM
> 
> 
> 
> Here are the memory registration modes, under the module 
> parameter "Memreg". The idea is they go faster as the setting 
> increases:
> 
> 0 - no memory registration (all inline, via bounce buffers)
> 1 - memory registration per-rdma chunk (very correct but 
> slow) 2 - memory window bind per chunk (not supported by 
> OpenIB) 3 - memory window bind with async unbind (ditto) 4 - 
> registration with FMR (what I'm attempting to implement) 5 - 
> "persistent" memory registration (all physical)
> 
> Modes 0, 1, 2 and 4 were thought to be safe. Now I'm not so 
> sure about 4. Maybe implementing memory windows isn't looking 
> so bad after all! :-)
> 

Tom, 

Not sure we managed to convince you anything about FMRs. Anyway,
I would suggest, even just for the sake of performance evaluation to
try the following FMR approaches:
0a - this is the allegedly fastest: using FMR to consolidate list of
pages
2a - with fmr map/unmap - same behavior as MWs (this is what you called
4)
3a - with fmr pool - will be like async unbind.

By the numbering, I'm trying to be consistent with the security level,
lower num
means fastest, higher number means conservative security wise, but can
be 
slower.

I wouldn't be surprised if you end up finding 0a a win-win for both the
client
and the server. If you end up finding differently, then that may also be
interesting.
BTW, iSER only works this way, the RFC does not allow passing a 
chunk list as far as I know...

-Dror