[ofa-general] Re: IPoIB path caching

[Sean Hefty]

> What I have in mind is that IPoIB must not use cached IB path info.
> 
> If the IB stack has path caching which is in the default flow of 
> requesting a path record, it should provide an API (eg flag to the 
> function through which one does path query) to request a non cached path.

Argh!  This was the original design.  I believe the current design is a 
better approach.  The ULP shouldn't care whether the PR is cached or not 
- only that it's usable.

> The design I was thinking to suggest for IPoIB is to almost always use 
> this API since this policy makes the implementation consistent with the 
> decisions made by the network stack neighbour cache

This defeats one of the benefit of caching, which is using a single 
GetTable query, versus literally hundreds or thousands of Get queries. 
Consider that constant all-to-all communication using IPoIB between 1024 
ports, with a 15 minute ARP table timeout would hit the SA with close to 
600 queries per second.

I agree with Michael that it would be better for a ULP to invalidate 
cache entries.

While I agree that there's the potential for a problem, given that IPoIB 
has always cached PRs and no one has reported problems, I think we're 
overstating the likelihood of issues occurring in practice.  Even the SA 
caches the path data -- getting a PR from the SA doesn't provide any 
additional guarantees.

- Sean