[ofa-general] How to establish IB communcation more effectively?

[Or Gerlitz]

Davis, Arlin R <arlin.r.davis at intel.com> wrote:
> For a connection (socket connect, exchanging QP info, private data, qp modify)
> using uDAPL socket cm versus rdma_cm I get:
> socket_cm on 1Ge == ~900us
> socket_cm on IPoIB (mlx4 ddr) == ~400us
> rdma_cm on IB (mlx4 ddr) == ~2200us
> As you can see, the path record queries via rdma_cm add a substantial penalty.

Hi Arlin,

Just to make sure we're on the same page: both IPoIB and the RDMA-CM
use SA path queries (ipoib for the unicast arp reply, and rdma-cm for
rdma_resolve_route), going into details, things look like:

with the rdma-cm:

rdma_resolve_addr
          A --> *  ARP request (broadcast)
          B --> A ARP reply (unicast, before that B does SA path query)
rdma_resolve_route
          A does SA path query
rdma_connect
          A --> B CM REQ
          B --> A CM REP
          A --> B CM RTU

with the socket cm / ipoib:

socket connect
          A --> *  ARP request (broadcast)
          B --> A ARP reply (unicast, before that B does SA path query)
          A --> B TCP SYN (unicast, A does SA path query!)
          B --> A TCP SYN + ACK
          A --> B TCP ACK

Looking on the differences between the flows, we can see that --both--
flows have --two-- path queries, so the 400us vs 2200us difference
can't be related to that.So, is it possible that you have counted
rdma_create_qp in the rdma-cm accounting and didn't count
ibv_create_qp in the scm accounting?

Or.