[openib-general] IB routing discussion summary

[Jason Gunthorpe]

On Thu, Feb 15, 2007 at 11:39:49AM -0800, Sean Hefty wrote:

> I think this may allow establishing inter-subnet connections.  As an
> example of its usage:
 
I think you are right, this does contain enough information.

> a. Active side issues a PathRecord query to the local SA with SGID=local,
> DGID=remote.
> b. SA responds with PathRecord(s).
> c. Active side selects local PathRecord P1.
> d. Active side issues a PathRecord query to the remote SA using PathRecord 
> P1 to
> format the request: SGID, DGID, SLID, DLID, TC, FL, SL, etc.
> e. The remote SA responds with PathRecord(s).  The SA must ensure that 
> packets injected into the internetwork using P1 will route to the returned 
> records.
> f. Active side selects remote PathRecord P2.
> g. Active side validates that remote packets injected using P2 route to P1.

Let me add something:
   - Side A GRH.SGID = P2.DGID
   - Side A GRH.DGID = P1.DGID
   - Side A GRH.TC/FL = P2.TC/FL
   - Side A LRH.SLID = P1.SLID
   - Side A LRH.DLID = P1.DLID
   - Side A LRH.SL   = P1.SL

   - Side B GRH.SGID = P1.DGID
   - Side B GRH.DGID = P2.DGID
   - Side B GRH.TC/FL = P1.TC/FL
   - Side B LRH.SLID = P2.SLID
   - Side B LRH.DLID = P2.DLID
   - Side B LRH.SL   = P2.SL
[Side A information programs the active sides QP. Side B information
goes into the REQ and is sent to passive side that then uses it
directly to program the QP. The passive side never does a PR.]

Inverting the TC/FL source can avoid requirement g. When P1 is
produced it also generates a TC/FL that ensures the LIDs match for the
reverse direction. When P2 is created it does the same.

Here is the interesting bit: If the SAs support 'multiple router
paths' then they must have a small bit of global information which is
that packets entering router LID x,y,z on subnet Y appear on local
subnet router port A. Enough information is in the P2 request
to lookup in this table to learn the input port.

In forming P2 the SA will use that bit of global informtion to
restrict the router LIDs to one that ends up on port A. Once it
selects a router LID and CA LID it produces a TC/FL that ensures those
LIDs are selected by the local router port A.

Choosing the DGIDs as I did can allow for some multipathing via GID if
an implementation goes that way. [Though IMHO, doing this creates new
ugly problems, and doesn't solve the SLID selection issue.]

This changes the definition of a PR, the returned GRH fields are the
fields that the *remote* must send to produce the LIDs in the PR.

If you define TC/FL like this then I don't know what happens
when UD makes a PR query and uses the returned TC/FL in the local QP
configuration... It may actually be better to have a new query type
and have SA take care of it.

In this solution new semantics for PR are defined, it requires the SA
magic GID, and it co-opts the flowlabel/tc to solve the QP LID
matching issue. Michael is probably right and we should find a way to
ask IBTA for implementation guidance. Hopefully that can be done
swiftly..

Jason