[openib-general] Re: [PATCH] CMA and iWARP

[Caitlin Bestler]

Steve Wise wrote:
> On Thu, 2006-01-26 at 09:16 -0800, Caitlin Bestler wrote:
>> openib-general-bounces at openib.org wrote:
>>> Here is a comment on the specific CMA/IWARP patch:
>>> 
>>> The iwarp enhancements in the this patch save the each device's
>>> node_guid in the associated cma_device.  The assumption was that the
>>> iwarp device's node_guid would be the mac address for that device.
>>> Then, in cma_acquire_iw_dev(), the rdma_dev_addr pulled from the
>>> netdev device as a result of route lookup is used to find a cma_dev
>>> who's node_guid matches the rdma_dev_addr pulled from the netdev.
>>> 
>>> In ethernet terms, the netdev's dev_addr is used to find an
>>> appropriate cma device with a matching node_guid.
>>> 
>>> This is broken, however, for multi-ported devices (and for devices
>>> who have multiple mac addrs per port), since there isn't a concept
>>> of a port guid in IB (i assume, since the code doesn't have port
>>> guids).   I discussed this with tom, and we think the correct
>>> solution is for the device to promote mac addresses as gids.  Then
>>> for each port, the iwarp device will advertise its mac
>>> address(es) and populate the gid cache with these mac addresses.
>>> 
>>> Then we can change cma_acquire_iw_dev() to find the appropriate gid
>>> from the gid cache.  In fact, cma_acquire_dev() might not need to
>>> switch out to IB vs RNIC functions.  It can probably be mostly done
>>> with common code. 
>>> 
>>> Thoughts?
>>> 
>>> I can provide a patch for this soon, but I'd rather get the current
>>> CMA changes into the trunk, then post a delta patch from the
>>> trunk... 
>>> 
>> 
>> By definition iWARP is cleanly layered over IP. Therefore an iWARP
>> port is not a physical port but a logical one.
>> 
>> Management of physical ports is something that must be done
>> independently of RDMA software.
>> 
>> For example, if two physical Ethernet ports are teamed this is NOT
>> visible to the RDMA layer. 
>> 
>> This is a major example of the need to let each transport express
>> itself naturally, and finding the common ground that is meaningful to
>> applications, rather than forcing one to emulate the other.
>> 
> 
> I'd like us to focus on phase I of iwarp.
> 
> For phase I, the CMA tries to find an appropriate openib
> device, given the dev_addr from the associated netdev that
> was found during a routing lookup.  For IB, the dev_addr is
> matched against gids.  For iwarp, the dev_addr is matched
> against the mac addr of the openib dev.
> 

Stop right there.

Once you have the associated netdev you should have 0 or 1
associated iWARP RNICs.

If you go any deeper you risk breaking existing solutions
for IP Aliasing, Ethernet teaming, etc.