[ofa-general] [RFC] host stack IB-to-IB router support

[Yaron Haviv]

> -----Original Message-----
> From: general-bounces at lists.openfabrics.org [mailto:general-
> bounces at lists.openfabrics.org] On Behalf Of Sean Hefty
> Sent: Monday, March 19, 2007 5:29 PM
> To: general at lists.openfabrics.org
> Subject: [ofa-general] [RFC] host stack IB-to-IB router support
> 
> Based on previous e-mail threads, this is my plan for implementing
IB-to-
> IB
> router support in the host stack capable of supporting RC
communication.
> Note
> that this work is part of the PathForward project aimed at supporting
> early
> IB-to-IB router development.  It is not intended to define IB router
> architecture.
> 
> 1. Extend struct ib_cm_req_param:
> 
>        struct ib_cm_req_param {
> 		struct ib_sa_path_rec	*primary_path;
> 		struct ib_sa_path_rec	*alternate_path;
> 	+	struct ib_sa_path_rec	*remote_primary_path;
> 	+	struct ib_sa_path_rec	*remote_alternate_path;
> 
>    The remote path information would be valid only if the provided
paths
> had
>    a hop_limit > 1, but could also be used to support paths where
>    reversible = 0.
> 
> 2. Add an ib_remote_sa module.
> 
>    This module would be responsible for obtaining remote path
information.
>    Because the architecture does not define how this information is
> obtained,
>    my intent is to encapsulate this functionality into a single module
to
>    simplify out of tree maintenance.  Its basic operation is:
> 
>    a. Local ib_remote_sa sends query request to remote ib_remote_sa.
>    b. Remote ib_remote_sa queries its local SA.
>    c. Remote ib_remote_sa sends query response to local ib_remote_sa.
> 
>    I expect the ib_remote_sa implementation to be a temporary solution
> only.
>    It will layer above either the ib_mad or ib_cm services, whichever
ends
>    up being easier.
> 
> 3. Extend the rdma_cm route resolution to include remote route lookup.
> 

Sean,

I believe a much simpler and more generalized solution would be to
imitate the behavior of IP routing rather than using the "remote sa"
concept 
I understand that you tried to work within the iba spec boundaries
(bugs), but those can change if needed 

An example would be:
1. client wants to open connection to DGID xx:yy (xx=Gid prefix) 
2. CM/SA layers check if xx=local subnet, if not use a local routing
table to map DGID to router DGID(xx) (can even lavarage on Linux IPv6
tables I assume)
3. issue path query to the router DGID 
4. message is sent to destination 
5. destination router either have the DGID->DLID/Path cached or if not
handles an exception and lookup the path(DGID) (just like any IP router)
6. router send to destination
7. destination conducts a reverse lookup 
...

You can notice the above is just like the routing process in IP/Eth 

A path is a local attribute of a subnet/L2, much like a MAC address, a
remote node shouldn't have visibility into L2 attributes of the remote
subnet, such a model is not robust, not going to sustain failures very
well, and can't scale

We need to learn from the IP world and generations of experience rather
than invent our own mechanisms 

Yaron   

> - Sean
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-
> general