[openib-general] gen2/rnic-pi differences

[Caitlin Bestler]

On 7/1/05, Fab Tillier <ftillier at silverstorm.com> wrote:
> > From: Caitlin Bestler [mailto:caitlinb at siliquent.com]
> > Sent: Friday, July 01, 2005 9:15 AM
> >
> > RNIC-PI defined several verb layer features that if supported
> > eliminate the need for a DTO_COOKIE. If the information can
> > be integrated with existing verb layer structures it is a
> > major improvement in efficiency, at worst case it merely
> > requires the verb layer to implement the same workarounds
> > that the Access Layer is already forced to use.
> >
> > These features are:
> >       all verb layer objects have a consumer supplied
> >       identifier (os_data) that is used to identify that
> >       object back to the consumer in all completions and
> >       callbacks. So instead of getting the QPID you get
> >       the EP pointer (assuming that is what you supplied).
> 
> This should be really easy to implement for Mellanox HCAs - the mthca driver
> already has to resolve the QP structure when processing completions, and getting
> the user's QP context and including it in the work completion should be a
> trivial addition (for someone familiar with the code base).
> 

That was the basic assumption, that the verb layer drivers
could implement these features more easily than DAT/IT-API
could because they would simply require fields in
existing data structures rather than totally separate
parallel data structures.

> >       Three flags are identified per work request that can
> >       be ignored, passed through or fully implemented. They
> >       are Local Solicited, Consumer0 and Consumer1. If 'Local
> >       Solicited' is defined it means that completion of the
> >       work request should be treated as though it were a
> >       solicited event (i.e., it qualifies for 'next solicited
> >       event' callback notification).
> 
> Would these flags be returned in the work completion?  I don't know if I quite
> understand what you're requesting here.  Do Consumer0 and Consumer1 represent
> bits in a flags field?
> 

Yes. The three flags end up being pass-throughs to the
work completion, although the Local Solicited bit can be
merged with the Remote Solicited bit without harm (you
don't really need to know which end set the bit).

> In the Mellanox HCA implementation, the 64-bit work request ID is stored by the
> driver and recovered upon a completion.  Basically, the HCA driver maintains
> DTO_COOKIE-like information for each work request already.  Due to this lookup
> requirement, the information stored per work request could be arbitrarily large
> if so desired.  I don't know if that holds true for the PathScale HCA hardware,
> though - if it doesn't it would require an additional lookup in the HCA driver
> to get back to this information, in effect pushing the DTO_COOKIE concept into
> the HCA driver rather than leaving it in the consumer.
> 

Exactly. Any implementation that has access to the original send
queue when reaping the work completion can easily extend the
64-bit work request ID with additional data.

I suspect that this is a very common implementation because
there isn't much purpose in actually sending the 64-bit ID out
over the system bus just to have it be shipped back in the
completion. Why pay for storage on the card to hold the 64-bit
wr id for every pending work request? And why pay to ship
it across the system bus twice?