[openib-general] SA cache design

[Sean Hefty]

Eitan Zahavi wrote:
> [EZ] Having N^2 messages is not a big problem if they do not all go one
> target... 
> CM is distributed and this is good. Only the PathRecord section of the
> connection establishment is going today to one node (SA) and you are
> about to fix it...

I expect that we'll start having issues scaling when the number of nodes starts 
to exceed the size of the CM's QP.  Your idea below should help.

> During initial connections setup you will not have anything in the SA
> cache and thus the SA will need to answer N^2 PathRecords. Smart
> exponential back-off can resolve that DOS attack on the SA at bring-up.

I'll post the code for the cache once I complete my testing, but it issues a 
single query to fill the cache.  The SA will only see O(n) requests.  The cache 
also supports an update delay, or settle time, and minimum update time to 
prevent spamming the SA with back to back requests.

> [EZ] We might need a little more in the key for QoS support (to come).

This would need to be exposed through our APIs as well.  Alternate paths are 
also not yet supported.

> [EZ] I would try and make sure the connections are not done in a manner
> such that all nodes try to establish connections to a single node at the
> same time. This is an application issue but can be easily resolve.

I agree.

> [EZ] I think a centralized CM is a only going to make things worse.

It can reduce the number of messages on the network from O(n^2) to O(n).  The 
idea is that instead of all nodes sending connection requests to all other 
nodes, they send a single connection request -- containing an array of QP 
information -- to one node.  (The array could be sent over an established 
connection, rather than in MADs.)  The amount of traffic to that one node should 
be only slightly worse than the all to all case.

- Sean