[openib-general] Problem is routing CM REQ

[Jason Gunthorpe]

On Fri, Feb 09, 2007 at 04:45:29PM -0500, Hal Rosenstock wrote:

> >    Off hand I don't see that the existing path record query structure
> >    has enough information to do this.. Particularly, in cases
> >    where each subnet has more than 1 router port there is no real
> >    guarentee that querying for the SGID -> DGID direction and then the
> >    DGID -> SGID direction uses the same router ports without providing
> >    both router LIDs as part of the query.
> 
> Router LIDs rather than GIDs (in the case of LMC > 0) ?

Yes, it is the router LID that is matched by the QP, the router GID
never makes it into any packets or PR responses. To elaborate on it..

Basically you need to specify the egress subnet *and* the egress
router LID when constructing the path to handle the case of multiple
fabric and router paths. The GID of the target and the LID of the
target's router port is enough disambiguate all the possible
multipaths down to a set that will match the QP programming.

This is all because of the LID matching rules. The ultimate router
egress LID must be controlled when establishing the path. It must
match the DLID in the QP, so it must be specified when the path is
looked up so that the SA/Routers/etc can provide a PR that meets the
egress LID requirement. This is not just to ensure that the router
selects the right LRH.SLID in the case of LMC >0 but to also ensure
that the *right* router port is used in the case of multiple
(redundant) routed paths.

Basically the idea where each end of a RC QP could independently do a
Path Record query for the remote GID cannot work due to the LID
matching rule. 

Sean: Even if you can query both SA's there isn't enough information
to force things to use the same router path in each direction.

Jason