[ofa-general] RE: [RFC] host stack IB-to-IB router support

[Sean Hefty]

>Would it become part of openfabrics or just as a 3rd party
>patch that interested parties could apply?

Portions of the changes should be suitable for upstream submission.  The
ib_remote_sa module could be added to an OFED release if there was enough
demand.

>So the idea is that the CM REQ now uses remote_*_path to set the
>fields? You intend to go with the notion that IBA specifies the
>active sides sets the passive's LIDs in all cases?

Correct - this keeps the CM compliant with what's defined in the spec.

>> 2. Add an ib_remote_sa module.
>
>It looks like there is a new wire protocol to support this?

Basically.  I would likely just exchange SA GET/GET response formatted MADs
between ib_remote_sa's, but use a vendor defined MgmtClass.

>Does it still work if, for instance, a modified active side tries to
>talk to an arbitary existing target (such as storage or something)?

The passive side must handle routed requests (include the GRH when forming the
AH for the response), but, yes, I would expect this to work.

One of my reasons for separating the ib_remote_sa into a separate module is that
allows running it on an arbitrary node on the remote subnet.  This should
support unmodified targets.   Additional work would just be needed to locate the
remote ib_remote_sa service.  (My initial implementation will just expect to
find the ib_remote_sa service running at the DGID.)

I would consider a slightly different design if this were taken too far,
however.  For example, issuing the query might belong in the kernel, but
responding to the query might belong in userspace.  It would depend on the
evolution of the architecture.  For now I was thinking of combining the
functionality simply for ease of implementation.

>Broadly, do you describe having the active side send a seperate GMP to
>the passive node to do PR queries and then sending a CM REQ?

Yes.

>> 3. Extend the rdma_cm route resolution to include remote route lookup.
>
>This means produce a ib_cm_req_param using ib_remote_sa when
>appropriate right?

Yes

>How does this compare to the idea of using the LIDs from the REQ's LRH
>on the passive side?

If I'm understanding this idea correctly, the LIDs from the REQ's LRH replace
the primary_local_lid and primary_remote_lid fields carried in the REQ.
Assuming that the other path fields in the REQ are usable, this is a much
simpler approach.  (This could be done by the ib_cm itself, or a loadable module
that snooped CM REQ messages.  The latter would avoid changes to the CM code if
that mattered.)

I can't really evaluate the trade-offs without an idea of how long such a
temporary solution will be needed, and how much functionality it needs to have.
Replacing LIDs in a REQ is easy enough that I can just do that while working on
a more flexible solution.

My proposal stays spec compliant and could be extended to support unmodified
targets, but requires a more defined wire protocol.  The drawback to extending
it is that we push further into non-architected areas.

- Sean