[ofa-general] [PATCH] rds: use separate ports for TCP and IB

[Jon Mason]

On Mon, Jun 02, 2008 at 10:29:22AM -0500, Steve Wise wrote:
> Olaf Kirch wrote:
>> On Friday 30 May 2008 18:29:01 Talpey, Thomas wrote:
>>   
>>> Yes, at the moment TCP to the iWARP NIC is like talking to a different
>>> host. But, RDMA-aware versions of a given protocol still need a second
>>> port, unless there is explicit upper layer support for initiating the MPA
>>> exchange. We have the same issue with NFSv3/RDMA, and we have
>>> applied for a second port (the application is still pending within IANA).
>>>     
>>
>> This may also become a problem for RDS, but in a different sense.
>> If you bind an RDS socket to a specific IP address, this also selects
>> a transport. If you bind to an IP address owned by ib0, you select
>> the IB transport. If you bind to an IP address owned by say eth0, you
>> select the TCP transport.
>>
>> Now if the one and only NIC in the system is an iWARP NIC, which transport
>> should we select? Both iWARP and TCP can be legitimate choices, depending
>> on the hosts we're talking to (and how do we find out whether the remote
>> node supports iWARP or not?)
>>   
>
> If you use different ports for TCP vs iWARP, then you could try one, and  
> if you don't connect (ie get a "connect refused") then you could try the  
> other one...

The only way I can see this working is if RDS serially goes through each IP
address on the system and tries to connect (and fails over to the next method
if unable).  Otherwise, it could destroy the current seperation that exists
in RDS between the IB and TCP modules.

>> I think for now it's okay to default to iWARP and punt if the remote
>> doesn't support it. But in the long run this is something that needs
>> to be addressed - either in RDS, or in the way ofed handles iWARP NICs.
>>
>> In fact it may not be such a bad idea to treat an iWARP NIC as two
>> devices, and register one ethX device (owned exclusively by the normal
>> stack) and one ibX device (owned mostly by the ofed stack, maybe with
>> the exception of ARP and such). 
>
> That is basically what happens in linux today.  There is a ethX  
> interface that plugs into the stack, and the rnic driver also registers  
> its rdma device with the rdma core.  RDS can detect this by virtue of  
> the fact that the MAC address is used as the gid for the device.  So if  
> a given ethX interface has an associated gid registered in the gid pool,  
> then you know that interface supports both TCP and iWARP.  Does that  
> make sense? 
>
> The rdma-cm code in core/cma.c does this sort of stuff for address  
> resolution for iwarp devices.  IE when resolving a remote address, the  
> rdma-cm looks up the remote addr in the kernel routing table.  That  
> returns a local net device, aka ethX.  The rdma-cm then looks up the mac  
> address for that device in the gid cache to see if that device is rdma  
> capable...
>
> Steve. 
>
>
>
>> Would that work with protocols like
>> NFS/RDMA neogtiation - ie can you negotiate specify a secondary port
>> _and_ address?
>>
>> Olaf
>>   
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general