[openib-general] openib gen2 architecture

[Roland Dreier]

    shaharf> It seems to me that the major design approach is to do
    shaharf> everything in the kernel but let user mode staff access
    shaharf> to the lower levels to enable performance sensitive
    shaharf> applications override all kernel layers. Am I right?

No.  The reason everything is in the kernel now is that we simply have
not started implementing any userspace verbs support.

    shaharf> I wonder if this is not the right time to come up with
    shaharf> much better abstraction - for user mode and for kernel
    shaharf> mode. For example, it seems that the abstraction layer
    shaharf> should abstract the IB networking objects and not the IB
    shaharf> hca interface. In other words - why not to build the
    shaharf> abstraction around IB networking types - UD, RC, RD,
    shaharf> MADS? Why do we have to expose the memory management
    shaharf> model of the driver/HCA to upper layers? Do we really
    shaharf> want to expose IB related structures such as CQs, QPs,
    shaharf> and WQE? Why? Not only that this is not good for
    shaharf> abstraction meaning changes in the drivers will require
    shaharf> upper layers modifications, but also this is very
    shaharf> problematic due security and stability reasons.

Keep in mind that CQs, QPs and other IB transport objects are
themselves abstractions.  I'm not opposed to better abstractions in
principle, but I think that the current level is a good one.  Any IB
hardware is likely to be optimized for implementing these
abstractions, and I have a hard time believing we are smart enough to
build layers on top of them that are both generic enough and efficient
enough for all applications.

    shaharf> I think that using correct abstraction is very critical
    shaharf> for a real acceptance in the Linux/open source world.

I agree.  However, I think the Linux kernel community will actually be
opposed to extra abstraction layers that hide what the hardware is
really doing.  For example, I believe any really high-performance IB
application is going to understand the work queueing model and want to
deal with QPs and CQs.

    shaharf> Why should we develop complicated functionality such as
    shaharf> RMPP in the kernel when the only few kernel based queries
    shaharf> (if any at all) will use them?

Funny you should raise this point.  I said the same thing some time
ago and Yaron Haviv violently disagreed ;)

 - Roland