[openib-general] SA cache design

[Sean Hefty]

I've been given the task of trying to come up with an implementation for an SA 
cache.  The intent is to increase the scalability and performance of the openib 
stack.  My current thoughts on the implementation are below.  Any feedback is 
welcome.

To keep the design as flexible as possible, my plan is to implement the cache in 
userspace.  The interface to the cache would be via MADs.  Clients would send 
their queries to the sa_cache instead of the SA itself.  The format of the MADs 
would be essentially identical to those used to query the SA itself.  Response 
MADs would contain any requested information.  If the cache could not satisfy a 
request, the sa_cache would query the SA, update its cache, then return a reply.

The benefits that I see with this approach are:

+ Clients would only need to send requests to the sa_cache.
+ The sa_cache can be implemented in stages.  Requests that it cannot handle 
would just be forwarded to the SA.
+ The sa_cache could be implemented on each host, or a select number of hosts.
+ The interface to the sa_cache is similar to that used by the SA.
+ The cache would use virtual memory and could be saved to disk.

Some drawbacks specific to this method are:

- The MAD interface will result in additional data copies and userspace to 
kernel transitions for clients residing on the local system.
- Clients require a mechanism to locate the sa_cache, or need to make 
assumptions about its location.

I'm sure that there are other benefits and drawbacks that I'm missing.

- Sean