[openib-general] APM support in openib stack

[Sean Hefty]

> We want to have as little requirements as possible, and performance
> consideration(I asked you before which method is faster to setup IB
> connection),
> So what I am thinking of is:
> 	1. for static MPI job, don't use libibcm.so and librdmacm.so
> 	2. for dynamic MPI job, add to use libibcm.so only, no IPoIB is
> required.

I usually think of IB connections in terms of 3 steps:

1. Identify the remote node.
How do you identify the remote node, and how is that information obtained?

2. Obtain a path record between the local and remote node.
Today, there are two libraries capable of providing this, libibmad and 
librdmacm.  The userspace MAD library gives greater control, and would likely do 
so even if a libibsa were created.

3. Establish a connection.

IMO, out of band connections should should be prohibited by the IB stack, but 
this would likely break a lot of existing code.  The IB CM is the only agent 
capable of detecting stale and duplicate connections between nodes.  Without it, 
applications are more susceptible to data corruption.

- Sean