[ofa-general] [PATCH] rds: use separate ports for TCP and IB

[Steve Wise]

Olaf Kirch wrote:
> On Friday 30 May 2008 18:29:01 Talpey, Thomas wrote:
>   
>> Yes, at the moment TCP to the iWARP NIC is like talking to a different
>> host. But, RDMA-aware versions of a given protocol still need a second
>> port, unless there is explicit upper layer support for initiating the MPA
>> exchange. We have the same issue with NFSv3/RDMA, and we have
>> applied for a second port (the application is still pending within IANA).
>>     
>
> This may also become a problem for RDS, but in a different sense.
> If you bind an RDS socket to a specific IP address, this also selects
> a transport. If you bind to an IP address owned by ib0, you select
> the IB transport. If you bind to an IP address owned by say eth0, you
> select the TCP transport.
>
> Now if the one and only NIC in the system is an iWARP NIC, which transport
> should we select? Both iWARP and TCP can be legitimate choices, depending
> on the hosts we're talking to (and how do we find out whether the remote
> node supports iWARP or not?)
>   

If you use different ports for TCP vs iWARP, then you could try one, and 
if you don't connect (ie get a "connect refused") then you could try the 
other one...


> I think for now it's okay to default to iWARP and punt if the remote
> doesn't support it. But in the long run this is something that needs
> to be addressed - either in RDS, or in the way ofed handles iWARP NICs.
>
> In fact it may not be such a bad idea to treat an iWARP NIC as two
> devices, and register one ethX device (owned exclusively by the normal
> stack) and one ibX device (owned mostly by the ofed stack, maybe with
> the exception of ARP and such). 

That is basically what happens in linux today.  There is a ethX 
interface that plugs into the stack, and the rnic driver also registers 
its rdma device with the rdma core.  RDS can detect this by virtue of 
the fact that the MAC address is used as the gid for the device.  So if 
a given ethX interface has an associated gid registered in the gid pool, 
then you know that interface supports both TCP and iWARP.  Does that 
make sense? 

The rdma-cm code in core/cma.c does this sort of stuff for address 
resolution for iwarp devices.  IE when resolving a remote address, the 
rdma-cm looks up the remote addr in the kernel routing table.  That 
returns a local net device, aka ethX.  The rdma-cm then looks up the mac 
address for that device in the gid cache to see if that device is rdma 
capable...

Steve. 



> Would that work with protocols like
> NFS/RDMA neogtiation - ie can you negotiate specify a secondary port
> _and_ address?
>
> Olaf
>