[openib-general] RFC MPI and app. requirements of OpenIB

[Sean Hefty]

>> To help clarify the trade-offs:
>>
>> The CMA allows the use of IP addressing for connection establishment and
>> abstracts device hotplug.  It also operates over any type of RDMA device.
>>
>> A disadvantage of using the CMA is that it may not select the best set of
>paths
>> between two or more nodes.
>
>What defines best ? Is this preference or disjointedness or something
>else ?

I was intentionally vague here to leave this up to the application developer to
define.  The application may decide that a particular path or set of paths is
better than another based on whatever criteria they choose.  The current CMA
provides less control over which paths are selected for connections than if the
user queried the SA for paths and selected one based on some algorithm.  (I'd be
surprised if an app actually did this though.)

>>   The IB CM also permits path failover on a single
>> HCA.  Use of the IB CM requires that clients also interface with the IB SA to
>> obtain path records.
>
>Note that interaction with the SA will be required for MPI when
>multicast groups are to be used.

An alternative is to provide UD and multicast/broadcast support in the CMA.  I
know that the Intel MPI runs over DAPL, which does not provide multicast
support.  Can MPI operate with unreliable multicast support?  Does MPI plan on
using IB multicast?

>> My personal recommendation would be for applications to use the CMA, but that
>> does result in losing some flexibility.
>
>Would the CMA ultimately support path failover ?

Only if there's enough demand.  Since IB failover is restricted to a single HCA,
I can see where a more robust failover mechanism would be desirable.

- Sean