[openib-general] configuring ipoib

[Grant Grundler]

On Fri, Oct 21, 2005 at 02:21:23PM -0400, Kanevsky, Arkady wrote:
> Thanks guys.
> Please, excuse my terminology.
> No I can route a single IP address to an IB port.

ok

> But how do I route 2 (or more) IP addresses to the same IB port?

You can associate multiple subnets with an interface and
indicate which IP address this host should respond to:

	ifconfig ib<n>:1 <ip>

e.g.
	ifconfig ib0 10.0.0.81 netmask 255.255.255.0
	ifconfig ib1 10.0.1.81 netmask 255.255.255.0
	ifconfig ib2 10.0.2.81 netmask 255.255.255.0
	ifconfig ib3 10.0.3.81 netmask 255.255.255.0
	ifconfig ib3:1 10.0.3.99 netmask 255.255.255.0

route -n output now looks like:
iowa:/usr/src/linux-2.6.13# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 ib0
10.0.1.0        0.0.0.0         255.255.255.0   U     0      0        0 ib1
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 ib2
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.0.3.0        0.0.0.0         255.255.255.0   U     0      0        0 ib3

iowa:/usr/src/linux-2.6.13# ifconfig ib3
ib3       Link encap:UNSPEC  HWaddr 00-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.0.3.81  Bcast:10.255.255.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:2044  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:128 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

iowa:/usr/src/linux-2.6.13# ifconfig ib3:1
ib3:1     Link encap:UNSPEC  HWaddr 00-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.0.3.99  Bcast:10.255.255.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:2044  Metric:1




> If I specify the same ib# it just changes an associated IP address for
> the port.

eh? Either you aren't getting it or are very persistent with the
"confusing" terminology. I'll try one more time in case it's the former.

Connect a linux box (e.g. laptop) to two subnets and configure both
NICs like normal for two distinct subnets. Disconnect the second cable.
Then from another box that has it's default route set to the first subnet,
try to ping the IP you "associated with the second port".
The linux box will respond.

E.g. I have two private subnets: 192.168.0.0/24 and 192.168.1.0/24.
Both are connected to my squid (http cache) server that also
provides NAT to the outside world. The squid server does NOT
do routing.

iowa:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0
owa:~# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.1 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.1 ms

--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.1/0.1 ms

Obviously, I can ping the gateway.
My point is the gateway will also respond to other subnets
that it owns:
iowa:~# ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=0.1 ms
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.1 ms

--- 192.168.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.1/0.1 ms
iowa:~# ping 192.168.0.11
PING 192.168.0.11 (192.168.0.11): 56 data bytes

--- 192.168.0.11 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

192.168.0.11 is alive and owned by a third machine
on that subnet. Just trying to demonstrate that the
squid server is NOT doing any routing.


> If I specify next ib# it returns an error since that ib# does
> not have a port behind it.

Right. No port means no ib#.

hth,
grant