[openib-general] Re: [PATCH] CMA and iWARP

[Steve Wise]

On Thu, 2006-01-26 at 09:16 -0800, Caitlin Bestler wrote:
> openib-general-bounces at openib.org wrote:
> > Here is a comment on the specific CMA/IWARP patch:
> > 
> > The iwarp enhancements in the this patch save the each
> > device's node_guid in the associated cma_device.  The
> > assumption was that the iwarp device's node_guid would be the
> > mac address for that device.
> > Then, in cma_acquire_iw_dev(), the rdma_dev_addr pulled from
> > the netdev device as a result of route lookup is used to find
> > a cma_dev who's node_guid matches the rdma_dev_addr pulled
> > from the netdev.
> > 
> > In ethernet terms, the netdev's dev_addr is used to find an
> > appropriate cma device with a matching node_guid.
> > 
> > This is broken, however, for multi-ported devices (and for
> > devices who have multiple mac addrs per port), since there
> > isn't a concept of a port
> > guid in IB (i assume, since the code doesn't have port guids).   I
> > discussed this with tom, and we think the correct solution is
> > for the device to promote mac addresses as gids.  Then for
> > each port, the iwarp device will advertise its mac
> > address(es) and populate the gid cache with these mac addresses.
> > 
> > Then we can change cma_acquire_iw_dev() to find the
> > appropriate gid from the gid cache.  In fact,
> > cma_acquire_dev() might not need to switch out to IB vs RNIC
> > functions.  It can probably be mostly done with common code.
> > 
> > Thoughts?
> > 
> > I can provide a patch for this soon, but I'd rather get the
> > current CMA changes into the trunk, then post a delta patch
> > from the trunk...
> > 
> 
> By definition iWARP is cleanly layered over IP. Therefore an
> iWARP port is not a physical port but a logical one.
> 
> Management of physical ports is something that must be done
> independently of RDMA software.
> 
> For example, if two physical Ethernet ports are teamed this
> is NOT visible to the RDMA layer.
> 
> This is a major example of the need to let each transport
> express itself naturally, and finding the common ground that
> is meaningful to applications, rather than forcing one to
> emulate the other.
> 

I'd like us to focus on phase I of iwarp.

For phase I, the CMA tries to find an appropriate openib device, given
the dev_addr from the associated netdev that was found during a routing
lookup.  For IB, the dev_addr is matched against gids.  For iwarp, the
dev_addr is matched against the mac addr of the openib dev.

Currently the phase I iwarp cma patch assumes a single mac address for
an openib iwarp device, mapped to the node_guid.  I'm proposing using
either the IB gid cache infrastructure to allow the iwarp openib device
to advertise its set of local mac addrs -or- make the iwarp cma code
search the list of port guids (roland pointed out that there are port
guids), and thus map the device's mac addr list to port guids.

This method parallels the current openib method for mapping IPoIB
netdevs to openib devices.  It is a loosely coupled model for
associating netdevs with openib devs (which I'm not particularly fond of
btw).  But it works, and we can make iwarp openib devs work in the same
manner.

I'm not sure that this method prohibits any of the issues you raise wrt
what a "port" is in iwarp.  But maybe I'm confused...


Steve.