[Ofvwg] Further thoughts on uAPI

[Liran Liss]

> From: ofvwg [mailto:ofvwg-bounces at lists.openfabrics.org] On Behalf Of Hefty,
> Sean

> 
> > The kernel common code side is pretty straightforward, just a bunch of
> > tables of function pointers, templates and idrs for each object_type.
> 
> This is the part where the intent and implementation are not clear to me.  I see
> value in generic user space handle validation, but I prefer they map to driver
> specific resources.  For example, I don't want a user space allocation of a QP to
> result in this:
> 
> 1403 struct ib_qp {
> 1404         struct ib_device       *device;
> 1405         struct ib_pd           *pd;
> 1406         struct ib_cq           *send_cq;
> 1407         struct ib_cq           *recv_cq;
> 1408         struct ib_srq          *srq;
> 1409         struct ib_xrcd         *xrcd; /* XRC TGT QPs only */
> 1410         struct list_head        xrcd_list;
> 1411         /* count times opened, mcast attaches, flow attaches */
> 1412         atomic_t                usecnt;
> 1413         struct list_head        open_list;
> 1414         struct ib_qp           *real_qp;
> 1415         struct ib_uobject      *uobject;
> 1416         void                  (*event_handler)(struct ib_event *, void *);
> 1417         void                   *qp_context;
> 1418         u32                     qp_num;
> 1419         enum ib_qp_type         qp_type;
> 1420 };
> 
> monstrosity being allocated in the kernel.  The size and layout of this sort of
> structure impacts scalability and performance.  This is where I want to make
> sure there's alignment on the overall architecture.
> 

How the kernel decides to represent objects is a kernel implementation issue, not an ABI issue.
As long as the ABI is expressive enough (e.g., pass only the relevant information for the typed object at hand), the kernel code could decide what to hold where.

> More specifically, I would switch the existing uABI commands to ioctl's, leaving
> the code paths mostly unchanged, but marking them as legacy.  Then add the
> new ioctls that you suggested, with future drivers hooking directly into the ioctl
> framework.  Honestly, I'm not sure the rdma core should define anything
> beyond the common ioctl structure.  Anything else can just be placed into the
> driver specific portion, or maybe rdma core sub-subsystem (e.g. ib, iwarp, roce,
> opa, mlx, ...)  Maybe we would define some common ioctl's across all devices,
> but I don't think they would look anything like the current commands.  They
> would need to be much higher level.

That's where the provider channel comes in.
For provider stuff, uverbs won't do anything else other than a way to get a channel + some conventions (maybe).
There won't be any matching kAPI for it.

For generic interfaces (currently includes Verbs, Ethernet QPs, and IB management), the new scheme should map what we have today in a flexible manner.
This would enable us, for example, to pass only RoCE addressing attributes while modifying a RoCE QP (and optionally optimizing the kernel representation as well).
These interfaces have a matching kAPI.

New object types could be supported with additional/alternative attributes to existing objects, where it makes sense.
The flexibility of the ABI would allow for both optimal parameter passing and kernel representation.
For example, if I want to add a counter that counts completions, I would add it as a new type of CQ. This would allow it to interoperate with the existing object model.
The same would apply for new transport types.

If a completely new generic concept is introduced, we can add additional interfaces and extend the kAPI to support it.

> 
> - Sean