[ofiwg] mapping adapter memory

[Hefty, Sean]

> The approach of "let the provider find the best way to use the hardware
> resources and hide that from the user" will not always work in this case,
> as the constraints on adapter-mapped memory may prevent it from being a
> general solution.  For example, it may be a limited resource, and so not
> every EP will have access to it, and those that do may have restrictions
> that other EPs do not have (e.g. fewer send credits available, smaller
> MTU).
> 
> Any thoughts on whether this is worth trying to come up with some
> abstraction for?  From Cisco's point of view, it definitely is - it
> requires some level of EP differentiation such that it cannot be totally
> transparent to the app, but it can drop latency by 20%, a worthwhile goal
> for us.

This seems like an issue where not all endpoints supported by a provider are the same, and gets into resource allocation management.  I'm not sure libfabric is the place to define resource management.

Conceptually, I can see where not all endpoints may have dedicated hardware behind them and may have to share resources with other endpoints, and potentially other processes.  Even adapters that can dedicate hardware resources to every endpoint may not perform well as a result of caching limitations on the HCA.  This could require an app to share resources (e.g. a kernel allocated QP) for specific communication channels.

Maybe a provider can expose some attributes on the 'optimal' use of any the underlying hardware, so that an application or job scheduler doesn't oversubscribe the hardware.  Reporting maximum values doesn't do that, since apps often allocate the max values expecting that there won't be any performance loss for doing so.

- Sean