[ofa-general] Re: IPoIB path caching

[Sean Hefty]

> I am willing to go with the local sa coming to serve large MPI jobs, so 
> you load as a prerequisite to spawning large all-to-all job.
> 
> But, I think the default for IPoIB needs to be usage of non cached PR.

I think this ties together two things that aren't directly related.  We 
have two network stacks running on top of each other here.  Their 
policies should be separate.

As an example, let's reverse this.  Imagine instead that you implement 
IB over IP.  Should an IB path refresh policy dictate that IP update its 
ARP tables?  Or, looking at it differently, do you prevent IP from 
updating the ARP table unless the IB stack asks for it?

The policy for local PR caching should be set by an administrator.  Now, 
we could provide a policy setting that ties it to the ARP cache, which 
sounds like a good idea.  This will be less efficient in some use 
models, more efficient in others.  But not all PRs belong to IPoIB, so 
we need a way to handle this.  However, I don't believe that we have to 
always enforce such a policy, especially since the current stack doesn't 
have this behavior today.

- Sean