[openib-general] Local SA caching - why we need it

[Woodruff, Robert J]

Michael wrote, 
>> 
>> We already had some discussion of this in Tampa. Local SA caching is
very
>> valuable for applications that use the rdma_cm. We already discussed
and agreed
>> to include it as one of the new features for OFED 1.2.  We have
already seen
>> that it provides benefits for scaling connection establishment using
Intel MPI,
>> which uses DAPL and rdma_cm.  Once other MPI's decide to move their
>> connection establishment model to use rdma_cm rather than the hack
they are
>> currently using, they will also need this feature for scalability.
If more
>> discussion is needed, we can certainly have it. 

>I guess having discussion in the open would certainly help.

>FWIW, things that IMO weren't fully addressed the last time this was
discussed
>and might be interesting to re-evaluate
>- what problem does the cache solve
>- what's the invalidation policy
>- how does it interact with complex SA queries and smart SA's

>There might be others.

>-- 
>MST

I suppose we should start a new thread for this discussion.
I have added Arlin and Sean, who have more details about the problems
that we have seen on a 256 node cluster with connection scaling
with OFED 1.1 and how the local sa cache helps solve the problem.
There was already one thread on this issue on the list, but
suppose we should have the discussion again.

I will let Sean and Arlin provide the details of why 
an SA cache is needed to allow connection establishment scaling to very
large clusters
since the SA can only handle a limited number of queries per second
and quickly becomes the bottleneck in trying to establish all-to-all
communications for MPI or other applications need all-to-all
communications. Intel MPI already sees this problem on a 256 node
cluster.
Other MPIs would see the problem, but are using a bad technique of using
sockets to exchange QP information and hard code connections, 
which has serious problems of it's own.

Arlin and Sean can provide the details.

woody