[ofiwg] mapping adapter memory

[Reese Faucette (rfaucett)]

> Conceptually, I can see where not all endpoints may have dedicated
> hardware behind them and may have to share resources with other
> endpoints, and potentially other processes.  Even adapters that can
> dedicate hardware resources to every endpoint may not perform well as a
> result of caching limitations on the HCA.  This could require an app to share
> resources (e.g. a kernel allocated QP) for specific communication channels.
> 
> Maybe a provider can expose some attributes on the 'optimal' use of any
> the underlying hardware, so that an application or job scheduler doesn't
> oversubscribe the hardware.  Reporting maximum values doesn't do that,
> since apps often allocate the max values expecting that there won't be any
> performance loss for doing so.

How about something like this:
There are hints the app can specify when creating an EP, such as "OPTIMIZE_FOR_LATENCY" or "OPTIMIZE_FOR_BW".  If the hardware has nothing special to do for those hints, they are ignored.  If there are some special hardware resources that can do one or the other, the provider will make a best-effort to match them with their respective hints.  Once these special resources are exhausted, "you get what you get".  So, if hardware supports N "low latency" QPs, and the app requests N+1 OPTIMIZE_FOR_LATENCY QPs, the N+1th QP will just be a little slower than the others.  Moral: ask for the more important ones first.

That seems not-to-invasive, and is effective for my needs.
-reese