[openib-general] Problem is routing CM REQ

[Jason Gunthorpe]

On Fri, Feb 09, 2007 at 12:58:51PM -0500, Hal Rosenstock wrote:
> > For simplicity, assume a single path.  My assumption in this case was that the 
> > SLID/DLID values would be reversed.  That is, the LIDs are relative to the local 
> > subnet, not the SGID.  But if I set the SGID = DGID = remote GID, then the LIDs 
> > would be relative to the remote subnet.  (Assuming that the local SA could 
> > support such a query at all.)
> > 
> > It seems that in order to meet the requirements of the spec, we need a way to 
> > perform inter-subnet queries.  (The alternative being to change the spec...) 
> > And if the local SA can return a path record to a remote DGID, then it also 
> > seems like the local SA must be able to collect some sort of information about 
> > the path to the remote subnet.  (How it does this seems TBD.)
> > 
> > So... I'm thinking that the solution to these problems should rest within the 
> > local SA...
> 
> Yes, this seems most consistent with what is there now although there
> are some issues to work out on how some of the fields are supported and
> which queries would work intersubnet (as well as how they would work).

I agree, some kind of inter subnet query will have to be used to make
this work consistently with the rest of IBA.

It looks to me like we overall need to have this look like:
- Routers need to be able to support inter-subnet reversible paths
  to meed the requirements for CM.
  - Inter-subnet reversible paths are defined to mean that when the LRH
    is selected on the destination subnet by the router it is reversible.
  - This can be signaled by using TClass and/or FlowLabel fields in the GRH.
- Routers need to be able to produce knowable SLIDs to meet the QP LID
  matching requirement
  - The LID to use can be signaled by using TClass and/or FlowLabel
- A kind of inter-subnet path record query is needed that can
  return a local and remote GRH and LRH. These four structures need to
  be *linked* so that:
   - Side A GRH.SGID = active side's Port GID
   - Side A GRH.DGID = passive side's Port GID
   - Side A LRH.SLID = any active side's port LID
   - Side A LRH.DLID = A subnet router
   - Side A LRH.SL   = SL to A subnet router

   - Side B GRH.SGID = Side A GRH.DGID
   - Side B GRH.DGID = Side A GRH.SGID
   - Side B LRH.SLID = any passive side's port LID
   - Side B LRH.DLID = B subnet router
   - Side B LRH.SL   = SL to B subnet router
   
   - When the A subnet router sees Side B GRH it produces
      LRH.SLID = Side A LRH.DLID
      LRH.DLID = Side A LRH.SLID
      LRH.SL   = SL to Side A Active side (may be != to Side A LRH.SL)
   - Similarly for Side B.      

   This linkage requirement is necessary due to the QP LID matching
   rules. I'm imagining that like SL the GRH.TClass and GRH.FlowLabel
   could be different in each direction.

   I'd think of this query as a generic duplex PathRecord query.

   Off hand I don't see that the existing path record query structure
   has enough information to do this.. Particularly, in cases
   where each subnet has more than 1 router port there is no real
   guarentee that querying for the SGID -> DGID direction and then the
   DGID -> SGID direction uses the same router ports without providing
   both router LIDs as part of the query.

Whatever responds to this query must be interacting with the router(s)
to ensure they recognize the GRHs and produce LRHs to meet all the
above requirements.

** The hackish and simple thing to do right now is to just demand that
   routers *always* use reversible LRHs with a single SLID and have the
   passive side pick up the QP lids from the LRH if it is routed..

Jason