[Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux

[Caitlin Bestler]

On 5/27/05, Bob Woodruff <robert.j.woodruff at intel.com> wrote:

> 
> So for every new RDMA device type that comes along, you need to add a new
> enum, and unions for device class specific stuff, etc.
> Seems rather static and not easily extended. Not
> to mention that testing nightmare when the thing has to support
> 20 different types of RDMA enabled devices.
> I think code like that could get pretty ugly pretty fast.
> 

It would not be every new device, it would be every new RDMA
verbs layer API. Gen2 already covers the needs of InfiniBand.
RNIC-PI covers the needs of InfiniBand, iWARP/TCP and iWARP/SCTP.

I would not anticipate a high rate of additional RDMA verbs 
appearing, particularly if the two existing verb APIs are unified.
Creating new verb APIs is something vendors do to enable new
markets, not because they have excess underutilized staff that
needs to  be kept busy.

Well, who knows what vendors with excess underutilized staff do,
there haven't been sightings of such creatures in memory.

But in any event, that can be dealt with in one of two ways:

struct rdma_xyz_ib {
};

struct rdma_xyz_iwarp {
};

union rdma_xyz_xport {
    struct rdma_xyz_ib ib;
    struct rdma_xyz_iwarp iwarp;
};

struct rdma_xyz {
   /* common fields */
   union rdma_xyz_xport xport;
};

Verus..

struct rdma_xyz {
     /* common fields */
};

struct rdma_xyz_ib {
    struct rdma_xyz common;
    /* ib fields */
};

struct rdma_xyz_iwarp {
    struct rdma_xyz common;
    /* iwarp fields */
};


The latter style is extensible, but makes it difficult to properly
allocate a buffer
that works for all variants.

The union style  is also already in use in both IT-API and RNIC-PI.

I personally prefer sub-classing to unions, but I have found myself in the 
minority on *most* projects where the issue has been discussed.
One reason is that sub-classing provides very little type-safety.
struct sockaddr is an example of this. It takes manual inspection
to ensure that the variants are properly differentiated and it is
still common for developers to pass in a plain struct sockaddr
without realizing that it is not large enough for a struct sockaddr_in6.


> I'd rather see a registration mechanism like what we already have
> with DAPL that does not require any code changes to add a new RDMA
> device/provider.  We have already proven that this works in DAPL
> as I know if at least 3 providers, IB, Myrinet, and RNIC (Ammasso)
> that were developed separately and were able to co-exist without
> any changes (enums and device class unions) in the DAT mid-layer.
> I assume that this can also be done with kDAPL in the kernel, but
> I defer to the DAPL experts to answer that one.
> 

There would be a registration method. And each device would load
a dynamic method table. What is in question is how many signatures
are supported.