[openib-general] openib gen2 architecture

[Sean Hefty]

shaharf wrote:

> It seems to me that the major design approach is to do everything in
> the kernel but let user mode staff access to the lower levels to
> enable performance sensitive applications override all kernel layers.
> Am I right?

The focus is only on kernel components at the moment, plus whatever 
user-mode support is needed to configure the fabric.

> It seems also that within the kernel, the ib interface/verbs (ib_*)
> is very close to the mthca verbs that are very close to vapi.

The agreement among the IB vendors was the start with VAPI as the base 
for the development of a new API.  But VAPI is little more than verbs as 
defined by the IB spec.

Infiniband hardware exposes PDs, CQs, QPs, etc, so I think it's natural 
for these constructs to appear in the software.  Abstractions away from 
IB specific constructs do exist in the form of SDP, ipoib, and DAPL.

> It seems to me that the current interfaces evolved to what they are
> today mainly because of the way IB itself evolved - with a lot of
> uncertainly and a lot of design holes (not to say "craters"). This
> forced most of the industry to stick with very straight forward
> interfaces that were based on Mellanox VAPI.

The verbs interface evolved because IB hardware is expected to expose 
this sort of functionality.  If you examine the layering of the 
software, ib_mthca and ib_core work together, with ib_core simply 
de-multiplexing among multiple devices.

> I wonder if this is not the right time to come up with much better
> abstraction - for user mode and for kernel mode.

I believe that we have the correct software layering.  At the lowest 
level you need software that talks directly with the hardware, with 
support for different hardware devices.  Abstractions, such as SDP and 
DAPL should be above that.  It seems that you're just wanting to move 
the abstraction down lower in the stack.

> Do we really want to expose IB related
> structures such as CQs, QPs,  and WQE? Why?

*blinks*

> etc.) requirements. I do not know if this requirement will actually
> realize, but if is will, the SM and maybe also the SMI/GSI agents and
> the CM will have to significantly change. If this is likely to

Coding requirements that may or may not happen or potential changes in 
the future is likely to produce nothing usable.

> Why should we develop complicated functionality such as
> RMPP in the kernel when the only few kernel based queries (if any at
> all) will use them?

I'm not opposed to moving functionality from the kernel to user-space, 
if it makes sense to do so.  Note that TCP is in the kernel, and RMPP is 
somewhat similar.

> very complicated interfaces and therefore much less
> stable and much more exposed to bugs, they will use 10GE.

Regardless of exposed interfaces, there will still be a need to 
implement IB management, meaning that the complexity will still be there.

> doubt. Yes, it is true that this project is meant to supply HPC code
> base, but eventually, IB will not survive as HPC interconnect only.

This is a debatable point.  I think that IB can survive as an only an 
HPC interconnect, and that it may have to.  Ethernet may never be as 
good as IB, but that doesn't mean that it won't someday be good enough, 
especially if it comes for "free" on the motherboard.