[ofa-general] Re: [PATCH RFC] RDMA/CMA: Allocate PS_TCP ports from the host TCP port space.

[Sean Hefty]

> Consider NFS and NFS-RDMA.  The NFS gurus struggled with this very issue 
> and concluded that the RDMA service needs to be on a separate port. Thus 
> they are proposing a new netid/port number for doing RDMA mounts vs 
> TCP/UDP mounts.  IMO that is the correct way to go:  RDMA services are 
> different that tcp services.  They use a different protocol on top of 
> TCP and thus shouldn't be handled on the same TCP port.  So, 
> applications that want to service Sockets and RDMA services concurrently 
> would do so by listening on different ports...

This is a good point, and a different view from what I've been taking. 
I was looking at it more like trying to provide the same service over 
UDP and TCP, where you use the same port number.  I just can't come up 
with any solution that works for iWarp, and sharing the port space seems 
like the only way to fix things.

> The iWARP protocols don't include a UDP based service, so it is not 
> needed.  But if you're calling it a UDP port space, maybe it should be 
> the host's port space?

I think it should match what's done for TCP.  IMO, there should be a 
connectionless RDMA service, along with multicast, over UDP/IP/Ethernet.  :)

> Yes.  The only exports interfaces into the host port allocation stuff 
> requires a socket struct.  I didn't want to try and tackle exporting the 
> port allocation services at a lower level.  Even at the bottom level, I 
> think it still assumes a socket struct...

I looked at this too at one point, and gave up as well.  I don't know 
what other assumptions are made in the stack as a result of this.  For 
example, if an app binds to an IP and port, and the IP address is 
removed and re-added, is the port still valid/reserved?

For iWarp, is using a struct socket essentially any different than 
transitioning an existing socket to RDMA mode?  You're just requiring it 
to be in a specific state.  Are there problems around doing this?  How 
much harder (technically, as opposed to politically) would it be to take 
this change a step farther and offload an active connection?

> I left it all in to show the minimal changes needed to implement the 
> functionality.  To keep the patch simple for initial consumption.  But 
> yes, the rdma-cm really doesn't need to track the port stuff for TCP 
> since the host stack does.

Okay - for final patches, I think we want to remove the rdma_cm specific 
port spaces, along with changing the API to clarify that it uses the 
same port space as TCP/UDP.

> I haven't looked in detail at the SDP code, but I would think it should 
> want the TCP port space and not its own anwyay, but I'm not sure.  What 
> is the point of the SDP port space anyway?

The rdma_cm needs to adjust its protocol for SDP over IB.  I'm not too 
concerned with SDP, since it's not upstream yet, but I don't want to 
break it beyond repair either.  The rdma_cm may not need to manage the 
SDP port space at all, and instead rely on SDP to ensure that it 
provides unique port numbers by itself.

- Sean