[ofa-general] RE: two interfaces with ipoib

[Or Gerlitz]

Sean Hefty wrote:
> Steve Wise wrote:
>> Guys, this reminds me of an issue we have with rnics and regular nics 
>> on the same physical network.  By default linux responds to arp 
>> queries on all ports it receives the query on.  This leads to very bad 
>> results with you're trying to do offloaded connections.  When 
>> resolving the address/route, the rdma client can end up getting the 
>> mac address of the dumb nic instead of the rnic.   I don't know if 
>> route resolution in the ib cm has this issue, but it might since they 
>> use ipoib for some part of the resolution, no?
> 
> I think this could be the problem.  (And could have taken me a long time 
> to figure it if it is, so thanks!)
> 
>> 2) If you have a multi-homed host and the physical ethernet networks are
>> bridged, then you need to configure arp to only send replies on the
>> interface with the target ip address:
>>
>>        sysctl -w net.ipv4.conf.all.arp_ignore=2

OK, Sean, sorry not to mention this to you, we have resolved this with a 
customer some time ago and I have communicated it to Mellanox on Sonoma 
  such that it will be added to the OFED 1.2 documentation.

Generally speaking, its a bad (somehow dead on arrival test for a system 
administrator) habit to have two IP (=L3) subnets sharing the same L2 
(specifically broadcast) domain. In infiniband (IPoIB) it means have two 
IP subnets over the same Partition and in Ethernet is means have two IP 
subnets over the same VLAN.

I understand the default setting of arp_ignore = 0 is a religious 
argument held once in a while at the netdev mailing list, if people from 
here want to try it, i am crossing my fingers for them, but again, it 
has nothing special to do with ipoib.

Or.