[ofa-general] Directions for verbs API extensions

[Steve Wise]

Roland Dreier wrote:
>  > > There are a few discrepancies between the iWARP and IB verbs that we
>  > > need to decide on how we want to handle:
>  > >
>  > >  - In IB-BMME, L_Keys and R_Keys are split up so that there is an
>  > >    8-bit "key" that is owned by the consumer.  As far as I know, there
>  > >    is no analogous concept defined for iWARP STags; is there any point
>  > >    in supporting this IB-only feature (which is optional even in the
>  > >    IB spec)?
>
>  > In fact there is an 8b key for stags as well. The stag is composed of
>  > a 3B index allocated by the driver/hw, and a 1B key specified by the
>  > consumer. None of this is exposed in the linux rdma interface at this
>  > point and cxgb3 always sets the key to 0xff.
>
> Oops, I completely missed that in the iWARP verbs spec.  Yes, the IB and
> iWARP verbs agree on the semantics here, so the only issue is that the
> "key" portion of L_Keys/R_Keys is only supported by IB devices that do
> BMME.  So we can expose this in the API without too much trouble.
>
>  > The chelsio driver supports the iwarp bind_mw SQ WR via the current
>  > API. In fact the current API implies that this call is actually a SQ
>  > operation anyway:
>  > > /**
>  > > * ib_bind_mw - Posts a work request to the send queue of the specified
>  > > * QP, which binds the memory window to the given address range and
>  > > * remote access attributes.
>  > 
>  > How is the current bind_mw API not valid or correct for iwarp MWs?
>  > Other than being a different call than ib_post_send()?
>
> That's the only issue.  The main impact is that you can't submit an MW
> bind as part of a list of send WRs.  I guess it's not too severe an
> issue.  I don't have any strong feelings here, except that eliminating
> the separate bind_mw call might be a little cleaner.  On the other hand
> it adds more conditional branches to post_send so maybe it's a net lose.
>   
BTW: looks like /usr/include/infiniband/verbs.h is missing a 
ibv_bind_mw() function.  The struct and context ops are there, but no 
API func.  This means there is no bind_mw support for user mode at this 
point.  So we don't have to worry about backwards compatibility...

Steve.