[ofa-general] Further 2.6.23 merge plans...

[Jason Gunthorpe]

On Tue, Jul 17, 2007 at 11:04:54PM -0700, Sean Hefty wrote:
> >IMHO, I still think that without some kind of SM/SA sourced
> >invalidation mechanism all client side caching (including the ipoib
> >stuff we have now) is a bad idea.
 
> Nothing precludes a user space daemon from updating the cache at
> timed intervals, or from communicating with an SA in some vendor
> defined way to maintain coherency.  I'm only trying to provide the
> kernel framework.  (We can debate whether another framework would
> have been better, and I've held this discussion on the list
> before...)  I do envision someone creating user space applications
> to control refreshes and, with local SA extensions, allow
> pre-loading of the cache, updates to specific paths, etc.

So, my main concern is with the role of kernel caching and especially with
how control is exported to user space.

Clearly the kernel needs a fast lookup cache for things like ipoib and
others. I don't think a kernel module needs or wants a full on
distributed SA.

I personally think a simple in-kernel (small) fast lookup cache merged
with the ipoib cache that has a netlink interface to userspace to
add/delete/flush entries is a very good solution that will keep being
useful in future. netlink would also carry cache miss queries to
userspace. In absense of a daemon the kernel could query on its own
but cache very conservatively. A userspace version of the very
agressive cache you have now could also be created right away.

This is because I firmly do not belive in caching as a solution to the
scalability problems. It must be solved with some level of replication
and distribution of the SA data and algorithms. With that view
pre-loading a gaint kernel cache is exactly the wrong kind of
user<->kernel interface.

Maybe you could summarise how the user/kernel interface works?  The
last I saw was something based on MADs that looked very inefficient
compared with netlink.

Jason