[openib-general][RFC]: CMA IB implementation

[Sean Hefty]

Guy German wrote:
> Basically I think that we can definitely agree that if the cma can 
> implement ib_at intended functionality it should replace it - no need to 
> have 2 modules doing the same thing.

I think that there will still be a need for a separate address translation 
module(s).  I have a call like the following in my latest version of the CMA:

int ib_resolve_addr(struct sockaddr *src_addr,
		    struct sockaddr *dst_addr,
		    void (*callback)(int status, struct ib_addr *addr,
				     void *context), void *context);

It uses ARP to convert a dst_addr and an optional src_addr to an sgid/dgid.  My 
intent is that this routine be moved to a separate module that deals only with 
resolving IP addresses to hardware addresses using ARP.

This essentially separates the ARP handling from ib_at into its own module.  I 
would rather start with some basic functionality that can be built upon rather 
than jumping directly to a ARP/ATS/QoS/caching handling interface.

> 1. Caching
> sean> generic SA caching module should be a part of ib_sa or exist 
> separately.
> 
> What about specific path records caching with event driven invalidate 
> logic ?

Caching will be complex, which is why I think that it needs to have its own 
module.  I'm envisioning a cache that can be saved to disk for faster system 
startup.

> 4. ATS registration
> sean> I think that ATS registration/deregistration should be integrated 
> with
> sean> IPoIB.
> 
> I don't think there is a consensus around that, but I don't know all 
> details.

This makes more sense to me than having the ATS code deference IPoIB private 
data structures.  However, if adding an rdma_ptr to the net_device can avoid 
this, then that will work.  And to be clear, I was referring to only 
registration/deregistration, not ATS queries.  It looks like the ATS code 
periodically scans all network devices in the system looking for changes in 
order to update the ATS records.  If this is the case, I would think that IPoIB 
should be able to do this more efficiently.  (I'm not that familiar with the 
code, so I could be off here.)

> 5. retries
> retries could be centralized in the ib_at approach.

For at least the ARP resolution module that I mentioned, I would expect all 
retries to be handled by that module.  For SA queries, since the MAD layer 
already performs retries, we just need to enable access through the API.

- Sean