<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Jason,<br>
<br>
Thanks for clearing up the use case. In that case doing ip_dev_find to
set oif would be wrong since it would not work correctly in the case
the same IP is associated with two devices. By just setting s_addr
before calling ip_route_output_key in addr_send_arp, that should take
care of it (the initial patch sent).<br>
<br>
>From what I can tell, this just fixes the policy routing case, without
affecting/addressing configurations that are using default routing. I
need to see why RDS/IB gets stuck in this case. My guess is that
hardware addresses don't get resolved correctly (as expected), and two
sides of an IB connection trip over a mismatch in what hardware a peer
thinks its using.<br>
<br>
But that is another issue that can be fixed independently. I'll add
some prints to see what might be happening.<br>
<br>
Thanks,<br>
Leo Tominna<br>
<br>
<br>
On 7/13/2009 6:49 AM, Jason Gunthorpe wrote:
<blockquote cite="mid:20090713134906.GA19062@obsidianresearch.com"
type="cite">
<pre wrap="">On Sun, Jul 12, 2009 at 08:38:38PM -0700, <a class="moz-txt-link-abbreviated" href="mailto:leo.tominna@oracle.com">leo.tominna@oracle.com</a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Associating the device with the source IP seems to be the correct thing to
do in general, but I initially avoided it in favor of source based routing
rules/tables since Linux does not do this by default. Source based routing
seems to be the only way to get load balancing right for regular IP
traffic
</pre>
</blockquote>
<pre wrap=""><!---->
Right, this is the Linux Way. The route table associated the output
device with the source/destination pair, and it is correct and
necessary to have source route policy entries to do what you are
talking about. As per the thread Or dug up older versions didn't do
this right - did it ever get fixed?
It is also necessary to use one of the arp_ignore settings otherwise
the receiver side responds with the wrong physical address. Source
routing fixes the transmitter, arp_ignore fixes the receiver.
</pre>
<blockquote type="cite">
<pre wrap="">when two local IPs are on the same subnet, so I thought it would be better
to have the src_ip alone cause the routing lookup to associate everything
with the correct device, as that would works for regular IP traffic and
anyone else, like ib_addr clients.
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
<blockquote type="cite">
<pre wrap="">The problems I was seeing with arp was that Linux associates arp entries
with specific devices, so if source based routing is used, and the arp send
does not take src_ip into account, the arp is sent from the default device,
and thats the device that gets the arp entry, where as neigh_lookup was
looking for an entry on the correct device, and was never finding the
neighbor.
</pre>
</blockquote>
<pre wrap=""><!---->
The proper flow is that the source/dest pair (+ plus extra) are used
to do a route lookup. The route lookup returns the output device to
use. The arp is sent out that device with source/dest pair. The
reciever will then receive the broadcast arp on all interfaces and you
need the arp_ignore setting to cause only the interface bound to that
IP to generate a response.
</pre>
<blockquote type="cite">
<pre wrap="">From what I can tell, on HPUX there is no device association with arp
entries. Does anyone know why Linux has this flexibility? It looks like
this was added in 2.2 or something, and I can't see any useful applications
for this.
</pre>
</blockquote>
<pre wrap=""><!---->
I can have multiple networks with overlapping IP spaces and using
routing tricks like policy routing I can generate overlapping network
specific ARP entries. Consider that I might have two ethernet ports
per machine, and two networks, both with the same IP. I can use policy
routing to direct all high priority traffic to one network, and low
priority to the other. The IPs are the same, but the macs are
different, thus the arp table must be keyed by interface and
destination.
Once you have complex routing stuff like policy routing this becomes
necessary, presumably HPUX does not have this..
</pre>
<blockquote type="cite">
<pre wrap="">The initial patch I sent is less restrictive but relies on source based
routing to get everything working, perhaps an explicit device mapping (as
above) makes more sense for RDMA traffic. Please correct me if something I
said is incorrect or if these changes conflict with other working
configurations.
</pre>
</blockquote>
<pre wrap=""><!---->
This is Linux, the RDMA IP behavior should exactly match the in-kernel
IP behavior. It wasn't clear to me if your patch did that or not?
On the surface it looks right, sending the arp from the device (and
IP?) the requesting packet is going out is the right thing..
Jason
</pre>
</blockquote>
</body>
</html>