[ofa-general] RE: pick the outgoing HCA based on the IP used for bind

[Richard Frank]

I played around with this a bit more yesterday - and it looks like 
rdma_bind_addr()->rdma_resolve_ip()->ip_dev_find() is always returning 
the first matching entry in the routing table... even though we are 
providing the source ip for the bind...

Keeping in mind that both IB ports have IPs on the same subnet...

[root at vosib8 rds-tools-1.1-2]# ip a s ib0
33: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast 
qlen 256
    link/infiniband 
80:00:04:04:fe:80:00:00:00:00:00:00:00:02:c9:02:00:20:3b:61 brd 
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 11.0.0.8/24 brd 11.0.0.255 scope global ib0
    inet6 fe80::202:c902:20:3b61/64 scope link
       valid_lft forever preferred_lft forever

[root at vosib8 rds-tools-1.1-2]# ip a s ib1
34: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast 
qlen 256
    link/infiniband 
80:00:04:05:fe:80:00:00:00:00:00:00:00:02:c9:02:00:20:3b:62 brd 
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 11.0.0.18/24 brd 11.0.0.255 scope global ib1
    inet6 fe80::202:c902:20:3b62/64 scope link
       valid_lft forever preferred_lft forever

[root at vosib8 rds-tools-1.1-2]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use 
Iface
11.0.0.0        *               255.255.255.0   U     0      0        0 ib0
11.0.0.0        *               255.255.255.0   U     0      0        0 ib1
10.10.0.0       *               255.255.255.0   U     0      0        0 eth3
42.2.0.0        *               255.255.255.0   U     0      0        0 eth2
139.185.139.0   *               255.255.255.0   U     0      0        0 eth1
10.12.0.0       *               255.255.255.0   U     0      0        0 eth0
169.254.0.0     *               255.255.0.0     U     0      0        0 ib1
default         whq2op-swi-1-rt 0.0.0.0         UG    0      0        0 eth1


Or Gerlitz wrote:
>
>     ucmatose allows binding to a specific address using -b.  I haven't
>     used rds-ping
>     to know if it's the same as -I in that case.  I don't have any
>     systems myself
>     with dual HCAs; I don't think they have enough slots to support
>     more than one.
>
>
> Hi Sean,
>
> ucmatose doesn't do anything with the address provided with the -b 
> param on its --active-- side, where this problem takes place. Yes, -I 
> to rds-ping is the same as -I to ping (other then the fact of the 
> former doesn't seem to work well). As I wrote you in detail, there's 
> no need for two HCAs to get the problem reproduced, just have one node 
> with two active port, each assigned with a different IP address.
>
> Or.
> ------------------------------------------------------------------------
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general