[ofiwg] mapping adapter memory

[Hefty, Sean]

This email is a little long, but here are my current thoughts on this topic. 

The domain attributes are expanded to include the following values:

ep_cnt - Total number of endpoints supported by the domain, subject to the specified or default endpoint attributes.  This is similar to a max_ep_count, but instead of being defined as a fixed value, it is a dynamic value.  If the provider has a fixed max value, it can use that here.  But if the number of endpoints depends on the EP attributes (such as the size), it would report a value relative to the associated attributes.

trans_ctx_cnt - The number of outbound command queues optimally supported by the domain.  For a low-level provider, this represents the number command queues to the hardware and/or the number of parallel transmit engines effectively supported by the hardware and caches.  Applications which allocate more transmit contexts than this value will end up sharing underlying resources.  By default, there is a single transmit context associated with each endpoint, but in an advanced usage model, an endpoint may be configured with multiple transmit contexts.

recv_ctx_cnt - Same as trans_ctx_cnt, but for the receive command queue.

ep_tc_cnt - Number of transmit contexts associated with an endpoint.  By default, this will be 1.  In a query, an application may request that the maximum value be returned, for example, by setting the input value to -1.

ep_rc_cnt - Same as ep_tc_cnt, but for receive contexts.

cq_cnt - Total number of completion queues supported by the domain, subject to the specified or default endpoint attributes.  Conceptually similar to ep_cnt.

cq_ctx_cnt - The number of completion contexts optimally supported by the domain.  Conceptually similar to trans_ctx_cnt and recv_ctx_cnt.  A CQ is associated with a single CQ context.  (It's not clear to me if this value is needed.)

The endpoint attributes are expanded to include these values:

trans_ctx_cnt - Number of transmit contexts to associate with an endpoint.  By default, this is 1.  Transmit contexts are independent command queues that may be configured separately, and are assumed to belong to different synchronization (locking) domains.  Transmit queues may be bound to separate CQs, with no ordering defined between transmit queues.  Applications may submit data transfers directly to a specific transmit context, or post to the associated endpoint, which allows the provider to select the transmit context.

recv_ctx_cnt - Same as trans_ctx_cnt, but for receives.  Receive contexts may be associated with a flow steering filter.  The filter would specify the criteria that incoming packets must match for the receive context to process it.  An endpoint would direct unmatched packets either to a default receive context, possibly using some algorithm.  Additionally, receive contexts may be targeted directly from the transmit side, using a simple index.

I think this exposes enough details in a generic enough fashion that a resource manager can figure out how to assign resources to processes optimally.  Simple apps ignore the transmit/receive contexts and only deal with the endpoint.  Low-level providers map contexts to command queues or HW contexts, and high-level providers can map contexts to processor cores or threads.

Latency and bandwidth concepts remain in the realm of endpoint QoS.

- Sean