[openib-general] RDMA CM and loopback addresses

[Jason Gunthorpe]

On Thu, Mar 30, 2006 at 10:45:29AM -0800, Sean Hefty wrote:
> Jason Gunthorpe wrote:
> >If you disable all packet filtering and you have two hosts
> >[10.0.0.1 and 10.0.0.2] doing the following on .2:
> >
> >ip route add 127.0.0.1 via 10.0.0.1 dev eth0
> >telnet -b 10.0.0.2 127.0.0.1
> >
> >And it will connect to .1's server. Turn rp_filter back on and it will
> >stop working again.
> 
> Thanks - I'll give this a try and see what happens.

You may have to do a bit more fiddling that just that to fully disable
127.0.0.1 as a local address on the source host, but it can be made to
work..

> >- Then consult the route table with a full tuple
> >  <src,dst,sport,dport,tclass,etc> to determine what device to send
> >  out on
> 
> Is it necessary to access the routing table twice?  With IB, once a source 
> address is obtained from the step above, it's mapped to a local IB device, 
> and the connection is established from that RDMA device.

Well, this is what happens in the normal IP stack. To match normal IP
semantics the source IP alone should never be mapped to a device, the
full tuple should be passed through the route table to get to a
device. This is what I was saying before, the IP is a property of the
host, not of a device.

Consider the following:

Host has two devices 10.2.0.2 and 10.0.0.1

ip addr add 10.0.0.1/24 dev ib0
ip addr add 10.2.0.2/24 dev ib1
ip route add default via 10.0.0.99 dev ib0 src 10.0.0.1

telnet 10.4.0.3  # Uses src = 10.0.0.1, outgoing through dev ib0
telnet -b 10.2.0.2 10.4.0.3  # Uses src = 10.2.0.2, outgoing through dev ib0

ip rule add from 10.2.0.1/26 table 32765
ip route add default via 10.2.0.99 dev ib1 table 32765 src 10.2.0.2

telnet -b 10.2.0.2 10.4.0.3  # Uses src = 10.2.0.2, outgoing through dev ib1

ip rule add to 10.4.0.0/24 table 32765

telnet 10.4.0.3  # Uses src = 10.2.0.2, outgoing through dev ib1

Ie this example uses policy routing to alter the outgoing device
selection based on the source or destination address. Things get even
more complicated if you include ip table NAT rules and netfilter..

So the first route lookup establishes the source address and the
second lookup would determine the outgoing device.

Just mapping the source IP to a device in a world were only on-link
prefixes are supported provides semantics that are close to the
default kernel established routes, but the admin can override those
routes to get different behavior.

> >For incoming I'd expect:
> >- Incoming connections *optimally* would include the src socket
> >  information so that that various policy mechanisms will work.
> 
> Source address information is provided, but to be clear, we're not using 
> sockets.

Ah I said socket just to imply the tuple <src, dst, sport ,dport,
tclass, etc> As long as the information is available on the wire then
at least the proper policy checks can be added in as time goes on.

Jason