[openib-general] SA cache design

[Hal Rosenstock]

Hi Sean,

On Tue, 2006-01-03 at 20:15, Sean Hefty wrote:
> Hal Rosenstock wrote:
> >>I've been given the task of trying to come up with an implementation for an SA 
> >>cache.  The intent is to increase the scalability and performance of the openib 
> >>stack.  My current thoughts on the implementation are below.  Any feedback is 
> >>welcome.
> >>
> >>To keep the design as flexible as possible, my plan is to implement the cache in 
> >>userspace.  The interface to the cache would be via MADs.
> > 
> > Would this be another MAD class which mimics the SA class ?
> 
> I hadn't fully figured this out yet.  I'm not sure if another MAD class is 
> needed or not.  My goal is to implement this as transparent to the application 
> as possible without violating the spec, perhaps appearing as an SA on a 
> different LID.

The LID for the (real) SA is determined from PortInfo:MasterSMLID so I
don't see how this could be done that way.

> >>  Clients would send 
> >>their queries to the sa_cache instead of the SA itself.  The format of the MADs 
> >>would be essentially identical to those used to query the SA itself.  Response 
> >>MADs would contain any requested information.  If the cache could not satisfy a 
> >>request, the sa_cache would query the SA, update its cache, then return a reply.
> >>
> >>The benefits that I see with this approach are:
> >>
> >>+ Clients would only need to send requests to the sa_cache.
> >>+ The sa_cache can be implemented in stages.  Requests that it cannot handle 
> >>would just be forwarded to the SA.
> > 
> > Another option would be for the SA cache to indicate what requests its
> > handles (some MADs for this) and have the clients only go to the cache
> > for those queries (and direct to the SA for the others).
> 
> I thought about this, but this puts an additional burden on the clients. 

Sure but how significant is this, especially if the 2 requests look
alike with some minor exception(s) like the class. I would think this
would make up for eliminating the extra indirection in the case where
the cache does not support the request.

> Letting the sa_cache forward the request allows it to send the requests to 
> another sa_cache, rather than directly to the SA.  There's some additional 
> flexibility that we gain in the long term design by forwarding requests.  (I'm 
> thinking of the possibility of having an sa_cache hierarchy.)

Sure; a hierarchial cache should scale even better.

> >>+ The sa_cache could be implemented on each host, or a select number of hosts.
> >>+ The interface to the sa_cache is similar to that used by the SA.
> >>+ The cache would use virtual memory and could be saved to disk.
> >>
> >>Some drawbacks specific to this method are:
> >>
> >>- The MAD interface will result in additional data copies and userspace to 
> >>kernel transitions for clients residing on the local system.
> >>- Clients require a mechanism to locate the sa_cache, or need to make 
> >>assumptions about its location.
> > 
> > Would SA caching be a service ID or set of IDs ?
> 
> I'd like the sa_cache to give the appearance of being a standard SA as much as 
> possible.

Yes, the closer to the real SA requests the cache requests are the
better.

>   One effect is that an sa_cache may not be able to run on the same 
> node as the actual SA,

Not sure why this would be the case.

>  but that restriction seems desirable to me.

Agreed.

> > Are there also issues around cache invalidation ?
> 
> I didn't list cache synchronization as an issue because I couldn't think of any 
> problems that were specific to this design, versus being a general issue.

Yes, this is a general design issue. The whole idea of how requests are
matched to the cache (what info is kept in the cache) and the
invalidation are keys. Just take PathRecords as one example.

-- Hal

> - Sean
>