[ofa-general] How to establish IB communcation more effectively?

[Sean Hefty]

>Yes, of-course. But, to start with, lets analyze the case of each node
>running --one-- rank and then take it from there to the case where
>each node runs C ranks.

The caching is independent of running MPI though.  To get a fair comparison,
you'd probably have to reboot the entire cluster before running the test and
ensure that no other communication between the nodes occurs over ipoib.

For myself, I'm not sure that the tests are the same.  The DAPL providers create
and modify the QPs differently.  I'd need to walk through the code to see
whether QP creation time is included and verify that the QP modify calls are the
same.

As for responding to the initial question, using sockets with hard-coded values
seems to be the most common way to establish IB connections at scale, though I
would guess that using the ib_cm with hard-coded values would work about the
same.
 
- Sean