[ewg] rdma/cm: disallow loopback address for iwarp devices

[Sean Hefty]

>Sean, can you try openmpi?  It fails for me, and yet ucmatose succeeds.
>I don't understand the difference yet...

I believe the issue is that rdma_bind_addr succeeds (returns 0), but no device
is assigned to the rdma_cm_id (verbs field is NULL).

This was a change from commit 6f8372b69c3198e06cecb1df2cb9682d0c55e657:

  The defined behavior of rdma_bind_addr is to associate an RDMA
  device with an rdma_cm_id, as long as the user specified a non-
  zero address.  (ie they weren't just trying to reserve a port)
  Currently, if the loopback address is passed to rdma_bind_addr,
  no device is associated with the rdma_cm_id.  Fix this.

There are two places where rdma_bind_addr() is called in the openmpi source code
(based on a tarball download of the trunk).  One is btl_openib_iwarp.c:

  rc = rdma_bind_addr(cm_id, ipaddr);
  if (rc || !cm_id->verbs) {
      rc = OMPI_SUCCESS;
      goto out3;
  }

The other is btl_openib_connect_rdmacm.c, but that deals with listening.  I
can't quickly determine if btl_openib_iwarp.c is usually used for IB or not.

So, to fully keep the behavior of 2.6.32, rdma_bind_addr for 127.0.0.1 should
succeed, but not assign a device.  I think this was the change from commit
..c55e657 that changed the behavior:

@@ -2089,7 +2096,9 @@ int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr
*addr)
        if (!cma_comp_exch(id_priv, CMA_IDLE, CMA_ADDR_BOUND))
                return -EINVAL;

-       if (!cma_any_addr(addr)) {
+       if (cma_loopback_addr(addr)) {
+               ret = cma_bind_loopback(id_priv);
+       } else if (!cma_zero_addr(addr)) {
                ret = rdma_translate_ip(addr, &id->route.addr.dev_addr);
                if (ret)
                        goto err1;

I'll see if reverting this gives the desired(?) behavior.

- Sean