[openib-general] Problem is routing CM REQ

[Sean Hefty]

> 1) What does the TClass and FlowLabel returned from SGID=local
>    DGID=remote mean?
>    Do you use it in the Node1 -> Node2 direction or the Node2 -> Node1 direction
>    or both?

Maybe it would help if we can agree on a set of expectations.  These are what I 
am thinking:

1. An SA should be able to respond to a valid PR query if at least one of the 
GIDs in the path record is local.

2. The LIDs in a PR are relative to the SA's subnet that returned the record.

3. An IB router should not failover transparently to QPs sending traffic through 
that router.

4. A PR from the local SA with reversible=1 indicates that data sent from the 
remote GID to the local GID using the PR TC and FL will route locally using the 
specified LID pair.  This holds whether the PR SGID is local or remote.

5. A PR from a remote SA with reversible=1 indicates that data sent from the 
local GID to the remote GID using the PR TC and FL will route remotely using the 
specified LID pair.  This holds whether the PR SGID is local or remote.

6. A PR with reversible=0 is relative to SA's subnet.  The SGID->DGID data flow 
over the PR TC and FL indicates the SLID->DLID mapping for that subnet.

Do your expectations differ from these?

The use of reversible between subnets is what's concerning me.  It may be that 
an SA could not return any paths as reversible between two subnets without using 
some trick like what you mentioned.

These add a requirement on the SA that they must be aware of the routes packets 
take between two GIDs using a given TC and FL, but I don't believe that this 
necessarily forces SA to SA communication.  The SA may only need to exchange 
information with a router...?

> Implicit in this are five IBA affecting things:
>  - that PRs with SGID=non-local mean something specific

I don't think that we're changing any of the meanings of the fields though.

>  - Routers do the SLID spoofing you outlined.

I'm not sure this is something that we do want now.  APM should really handle 
path failover.

> There is alot of complex work in the router and SA side to make this
> kind of topology work, but it is critical that the clients use path
> queries that can provide enough data to the SA and return enough data
> to the client to support this.

I'm still deciding if the existing path record attribute is sufficient.

- Sean