[openib-general] SA cache design

[Rimmer, Todd]

> From: Sean Hefty [mailto:sean.hefty at intel.com]
> Your kernel footprint is smaller than I expected, which is 
> good.
The key is that while there are O(N^2) path records in a fabric, only O(N) are of interest to a given node.  Hence if you only replicate entries where this node is the source the size of the replica is significantly smaller.  If someone is curious and wants to see all path records in the system, that would be a query you would let go through to the SA (and it would be a very infrequent query since no real world app, beyond fabric debug tools, would care about the paths which don't involve the node making the query).

This of course implies the "SA Mux" must analyze more than just the attribute ID to determine if the replica can handle the query.  But the memory savings is well worth the extra level of filtering.

>  Note that with
> a MAD interface, kernel modules would still have access to 
> any cached data.  I
> also wanted to stick with usermode to allow saving the cache 
> to disk, so that it
> would be available immediately after a reboot.  (My 
> assumption being that
> changes to the network topology would be rare, so we could 
> optimize around a
> stable network design.)
It is risky to assume that PathRecords would stay the same across a node reboot.  It is very likely that the SM could assign different LIDs or if the node is down for an extended period other things in the fabric could have significantly changed.

> 
> As a related topic, there will be a separate SA client 
> interface defined that
> will generate SA query MADs for the user.
Given the complexity of the RMPP protocol and the subtle bugs which everyone has encountered while implementing and debugging it (timeouts, retries, abort, window size management, class header offset, etc), it would be best to limit the number of copies of this protocol within the system.  Keeping the RMPP details hidden just in the kernel would be best.  An analogy might be the way sockets hides the details of the TCP/IP protocol from applications.  While I'm not aware of any changes in the works, we all remember the significant changes which occurred between IBTA 1.0 and IBTA 1.1 in the RMPP area.  If any similar significant revision to the protocol occurred it would be best to have it all implemented in just one place.

my $0.02

Todd Rimmer