[openib-general] Problem is routing CM REQ

[Hal Rosenstock]

On Fri, 2007-02-09 at 14:20, Jason Gunthorpe wrote:
> On Fri, Feb 09, 2007 at 12:58:51PM -0500, Hal Rosenstock wrote:
> > > For simplicity, assume a single path.  My assumption in this case was that the 
> > > SLID/DLID values would be reversed.  That is, the LIDs are relative to the local 
> > > subnet, not the SGID.  But if I set the SGID = DGID = remote GID, then the LIDs 
> > > would be relative to the remote subnet.  (Assuming that the local SA could 
> > > support such a query at all.)
> > > 
> > > It seems that in order to meet the requirements of the spec, we need a way to 
> > > perform inter-subnet queries.  (The alternative being to change the spec...) 
> > > And if the local SA can return a path record to a remote DGID, then it also 
> > > seems like the local SA must be able to collect some sort of information about 
> > > the path to the remote subnet.  (How it does this seems TBD.)
> > > 
> > > So... I'm thinking that the solution to these problems should rest within the 
> > > local SA...
> > 
> > Yes, this seems most consistent with what is there now although there
> > are some issues to work out on how some of the fields are supported and
> > which queries would work intersubnet (as well as how they would work).
> 
> I agree, some kind of inter subnet query will have to be used to make
> this work consistently with the rest of IBA.
> 
> It looks to me like we overall need to have this look like:
> - Routers need to be able to support inter-subnet reversible paths
>   to meed the requirements for CM.
>   - Inter-subnet reversible paths are defined to mean that when the LRH
>     is selected on the destination subnet by the router it is reversible.
>   - This can be signaled by using TClass and/or FlowLabel fields in the GRH.
> - Routers need to be able to produce knowable SLIDs to meet the QP LID
>   matching requirement
>   - The LID to use can be signaled by using TClass and/or FlowLabel
> - A kind of inter-subnet path record query is needed that can
>   return a local and remote GRH and LRH. These four structures need to
>   be *linked* so that:
>    - Side A GRH.SGID = active side's Port GID
>    - Side A GRH.DGID = passive side's Port GID
>    - Side A LRH.SLID = any active side's port LID
>    - Side A LRH.DLID = A subnet router
>    - Side A LRH.SL   = SL to A subnet router
> 
>    - Side B GRH.SGID = Side A GRH.DGID
>    - Side B GRH.DGID = Side A GRH.SGID
>    - Side B LRH.SLID = any passive side's port LID
>    - Side B LRH.DLID = B subnet router
>    - Side B LRH.SL   = SL to B subnet router
>    
>    - When the A subnet router sees Side B GRH it produces
>       LRH.SLID = Side A LRH.DLID
>       LRH.DLID = Side A LRH.SLID
>       LRH.SL   = SL to Side A Active side (may be != to Side A LRH.SL)
>    - Similarly for Side B.      
> 
>    This linkage requirement is necessary due to the QP LID matching
>    rules. I'm imagining that like SL the GRH.TClass and GRH.FlowLabel
>    could be different in each direction.
> 
>    I'd think of this query as a generic duplex PathRecord query.
> 
>    Off hand I don't see that the existing path record query structure
>    has enough information to do this.. Particularly, in cases
>    where each subnet has more than 1 router port there is no real
>    guarentee that querying for the SGID -> DGID direction and then the
>    DGID -> SGID direction uses the same router ports without providing
>    both router LIDs as part of the query.

Router LIDs rather than GIDs (in the case of LMC > 0) ?

The SA PathRecord may have room but the MultiPathRecord is pretty
tightly packed now.

-- Hal

> Whatever responds to this query must be interacting with the router(s)
> to ensure they recognize the GRHs and produce LRHs to meet all the
> above requirements.
> 
> ** The hackish and simple thing to do right now is to just demand that
>    routers *always* use reversible LRHs with a single SLID and have the
>    passive side pick up the QP lids from the LRH if it is routed..
> 
> Jason