[ofa-general] Re: [RFC] [PATCH 0/3] 2.6.22 or 23 ib: add path record cache

[Sean Hefty]

>Has anyone thought about using replication rather than caching to
>solve this problem? It seems to me it would be alot faster for some
>single process in the network to fetch and keep a copy of the entire
>SA route database, format it into a binary format and use RC RDMA to
>transfer it to every node each time it changes.

I have given thought to using RC RDMA to distribute the data to all nodes,
especially to eliminate the MAD protocol overhead.  There are a couple issues
with this:

To work with existing SAs, we need to working within the defined SA interface
(i.e. SA MADs), so something still needs to query for all path records.

The GetTable query requires an SGID, which means that whatever node collects the
path records must first collect all the GIDs.  (And the most efficient way I've
found to obtain a list of all GIDs is via a GetTable path record query...)  This
also means that the node collecting the path records will generate 1 query per
GID.  This has the same impact on the SA as each node issuing their own query.
And the impact on the subnet is higher, since we still need to distribute that
data to the end nodes.

In short, until we can standardize on some new SA interface, or we have a
distributed SA, I don't see where we can do much better than caching GetTable
responses at the end nodes.

- Sean