[ofw] [RFC] Locally generated path records

[Sean Hefty]

>>> To create a path record, IPoIB needs the following values (in addition
>>> to the ones it has access to for the AV creation):
>>
>> You don't need to create a PR, versus just creating the AV.
>
>I do if I want to give it to the CM to establish an RC connection.  Phase 2
>(creating PRs) is specifically to avoid the PR lookup for users of IBAT.
>Basically, rather than returning a GID pair, IBAT would return a path record.
>IPoIB would create that path record.  This effectively eliminates path queries,
>which should help things scale.

I thought you were only talking about using AVs for IPoIB, not having IPoIB
return PRs for someone else's use.

As long as there are separate interfaces for getting PRs - one that goes to the
SA, and another that goes to some magic PR generating entity, there shouldn't be
any harm.  An application that wants magical generated PRs just needs to be
aware of the limitations of pixie dust.

>Come to think of it, this affects AV creation too - to create a local AV
>without getting the path record means that you assume that the path is
>reversible (otherwise you can't use the SLID of a received packet as DLID for a
>send packet, can you?)

I was thinking that the generated AV would be reversible, but the message was
received as multicast and goes out as unicast...  So, this solution limits the
subnet topologies, but I'm not sure how big an issue that is in practice.  Both
the windows and linux IB CM's only support reversible paths.

>Looking at OpenSM, it always sets the rate to 12 (OSM_DEFAULT_SUBNET_TIMEOUT),
>both for MC groups as well as for path records.

This sounds more like packet lifetime, versus rate.

>I can trap that easily enough - if the subnet prefix is different, I can return
>a path that only has the GID/LID pairs filled in.  The CM code can then detect
>if everything else is zero, and issue a real path query.  While a bit
>convoluted, it avoids having to return PENDING from the IBAT library.

I wouldn't put path selection logic into the CM.  But I also wouldn't worry
about crossing an IB router at this point.  The current PR format doesn't work
anyway because you need LID information for both the local and remote subnets.

- Sean