[openib-general] Re: [PATCH] CMA and iWARP
caitlinb at broadcom.com
Thu Jan 26 10:27:00 PST 2006
Steve Wise wrote:
> On Thu, 2006-01-26 at 09:16 -0800, Caitlin Bestler wrote:
>> openib-general-bounces at openib.org wrote:
>>> Here is a comment on the specific CMA/IWARP patch:
>>> The iwarp enhancements in the this patch save the each device's
>>> node_guid in the associated cma_device. The assumption was that the
>>> iwarp device's node_guid would be the mac address for that device.
>>> Then, in cma_acquire_iw_dev(), the rdma_dev_addr pulled from the
>>> netdev device as a result of route lookup is used to find a cma_dev
>>> who's node_guid matches the rdma_dev_addr pulled from the netdev.
>>> In ethernet terms, the netdev's dev_addr is used to find an
>>> appropriate cma device with a matching node_guid.
>>> This is broken, however, for multi-ported devices (and for devices
>>> who have multiple mac addrs per port), since there isn't a concept
>>> of a port guid in IB (i assume, since the code doesn't have port
>>> guids). I discussed this with tom, and we think the correct
>>> solution is for the device to promote mac addresses as gids. Then
>>> for each port, the iwarp device will advertise its mac
>>> address(es) and populate the gid cache with these mac addresses.
>>> Then we can change cma_acquire_iw_dev() to find the appropriate gid
>>> from the gid cache. In fact, cma_acquire_dev() might not need to
>>> switch out to IB vs RNIC functions. It can probably be mostly done
>>> with common code.
>>> I can provide a patch for this soon, but I'd rather get the current
>>> CMA changes into the trunk, then post a delta patch from the
>> By definition iWARP is cleanly layered over IP. Therefore an iWARP
>> port is not a physical port but a logical one.
>> Management of physical ports is something that must be done
>> independently of RDMA software.
>> For example, if two physical Ethernet ports are teamed this is NOT
>> visible to the RDMA layer.
>> This is a major example of the need to let each transport express
>> itself naturally, and finding the common ground that is meaningful to
>> applications, rather than forcing one to emulate the other.
> I'd like us to focus on phase I of iwarp.
> For phase I, the CMA tries to find an appropriate openib
> device, given the dev_addr from the associated netdev that
> was found during a routing lookup. For IB, the dev_addr is
> matched against gids. For iwarp, the dev_addr is matched
> against the mac addr of the openib dev.
Stop right there.
Once you have the associated netdev you should have 0 or 1
associated iWARP RNICs.
If you go any deeper you risk breaking existing solutions
for IP Aliasing, Ethernet teaming, etc.
More information about the general