[ofw] [RFC] Locally generated path records

[Sean Hefty]

>To create a path record, IPoIB needs the following values (in addition to the
>ones it has access to for the AV creation):

You don't need to create a PR, versus just creating the AV.

>Reversible: Hard code to 1
>NumbPath: Hard code to 1

You shouldn't need these, at least for UD.  Reversible is needed if you end up
with IPoIB connected mode, but NumbPath is only used in a PR query.  If you want
to support any arbitrary topology with connected mode IPoIB, then you would need
to know if a path is truly reversible, and potentially want to use different
forward and reverse paths.

>PKey: Same as IPoIB port object
>MTU: broadcast group
>Rate: broadcast group

The rate seems to be the only real limitation to me.  In the worst case, you
slow down traffic between a given pair of nodes, but at least things keep
working.  Avoiding the PR queries seems like a good idea to me, but it should
probably be user configurable.

>Preference: 0

Only used for PR queries.

I'm not sure that most MPI apps will run through an IB router, so always
querying for off subnet paths will probably be needed.  (The current PR format
only works for UD traffic between IB subnets anyway.)

- Sean