[openib-general] ib_local_sa testing and observations.

[Sean Hefty]

The PathForward requirement is to provide userspace applications with all path 
records from the local system.  Whether we cache the paths locally or not, an SA 
needs to be able to handle this sort of request if it wants to support larger 
fabrics.  Unfortunately, I have no idea how large of a fabric openSM can support.

Without local caching, the hit on the SA can be substantially larger.  Each 
process (as opposed to node) will generate their own SA queries, which 
historically has been done using one query for each remote endpoint.  The 
caching is supposed to be there to help, but can be easily disabled if it is 
causing more traffic than it prevents.

Personally, I think that large fabrics will eventually need to be supported by 
some sort of distributed SM, even if they hide behind a single SM LID.

- Sean