[ofa-general] RE: [PATCH] Request For Comments:

[Sean Hefty]

Thanks for looking at this.

>Here is the top level API change I'm proposing for enabling interoperable
>peer2peer mode for iwarp.  I want to get agreement on how to expose
>this to the application before posting more of the gritty details of
>the kernel driver changes needed. The plan is to include this support
>in linux-2.6.27 + ofed-1.4.

I don't have a better idea what to call this, but when I think of peer to peer,
I think of that as the connection model, not a channel usage restriction.

>Does this require an ABI bump?

I'd like to avoid breaking the ABI or userspace API if possible.

>Note:  We could do this several ways.  I'm proposing one with this
>uncompiled patch.  The downside of my proposal is the applications have
>to change to turn this on.  However, I'm not sure thats too painful.
>We would have OMPI turn it on, and maybe even uDAPL so that all uDAPL
>ULPs would get it (IMPI, dapltest, HPMPI).

We could use rdma_set_option() for this.  If we do go the route of changing the
rdma_conn_param, adding generic flags or options would be more extendible.

>- always do peer2peer and don't let the app choose.  This forces
>the overhead of p2p mode on all apps, but preserves the API.

If we use rdma_set_option, I guess we could always enable it by default, and let
apps disable it.  I'm unsure if the better default is avoiding the overhead or
making the API easier to use, but I'm leaning toward the latter in this case.

>- use and environment variable that librdmacm will query.  This doesn't
>force p2p, and has the beneifit of not changing the API.  But at the
>expense of adding environment variables to the rdma-cm model.  This is
>used extensively in MPIs and even DAPL.  I think its an alternative
>we should consider.  This approach, however, doesn't help kernel
>applications.

I'm not thrilled with this idea.  Although I'm fine with the kernel solution
being different from the userspace one.

- Sean