[ofa-general] Re: [RFC] host stack IB-to-IB router support

[Jason Gunthorpe]

On Mon, Mar 19, 2007 at 05:12:42PM -0700, Sean Hefty wrote:

> >> 2. Add an ib_remote_sa module.
> >
> >It looks like there is a new wire protocol to support this?
> 
> Basically.  I would likely just exchange SA GET/GET response formatted MADs
> between ib_remote_sa's, but use a vendor defined MgmtClass.
> 
> >Does it still work if, for instance, a modified active side tries to
> >talk to an arbitary existing target (such as storage or something)?
> 
> The passive side must handle routed requests (include the GRH when forming the
> AH for the response), but, yes, I would expect this to work.

I think this is probably the best reason to doing something like
this. Even if existing targets don't work right out of the box the
changes to properly make a GRH should be fairly minor..
 
> One of my reasons for separating the ib_remote_sa into a separate
> module is that allows running it on an arbitrary node on the remote
> subnet.  This should support unmodified targets.  Additional work
> would just be needed to locate the remote ib_remote_sa service.  (My
> initial implementation will just expect to find the ib_remote_sa
> service running at the DGID.)

Hmm. If the goal is enable router development and experimentation then
it would be best if the 'ib_remote_sa' server was in user space, delt
with all 4 path records in one query and was centralized so it could
be made to store routing topology and configuration to solve the
multipath problems. Otherwise I think you are better to just talk
directly to the SA.

Maybe the best thing here is to have a simple ib_remote_sa client
module that just consults a list of servers and makes a normal SA
query. People working on multipath router support could then extend
that to specify a non-SA server and a new 4 path query type.

A list something like:
2001::/64 2001:1 SA
2001::/64 2001:2 SA
2002::/64 2000:1 not-SA   <-- On the local subnet.. new 4 PR format

Set via netlink or sysfs..

To start with no ib_remote_sa server would be needed, just a boot
script to set the expected SA addresses. You could define the MAD
format for a new 4 PR query but not implement a server to handle it.

> If I'm understanding this idea correctly, the LIDs from the REQ's
> LRH replace the primary_local_lid and primary_remote_lid fields
> carried in the REQ.  Assuming that the other path fields in the REQ
> are usable, this is a much simpler approach.  (This could be done by
> the ib_cm itself, or a loadable module that snooped CM REQ messages.
> The latter would avoid changes to the CM code if that mattered.)

My guess is that having a way for conformant 'unmodified' targets to
work is fairly important for alot of interesting applications at the
start. Otherwise using the LRH method is probably much better due to
the simplicity. I particularly like it because it can intrinsicly
support even the most complex routed environments, although without
APM/etc.

Do you have any idea what the PathForward program expects to do here?

Jason