[Ofvwg] Further thoughts on uAPI

[Hefty, Sean]

> Specifically, let's have generic ioctl of the form:
> 
>   create_object
>   query_object
>   modify_object
>   delete_object
> 
> Generically taking the same common struct:
> 
>   struct {
>      u32 length;
>      u32 object_type; // Eg QP,PD,etc
>      u32 user_handle; // in/out
>      u32 reserved;
>      u64 data[];
>   };
> 
> Where data follows some kind of structured attribute format like Liran
> was exploring ala netlink.
> 
> From this point we can obviously capture nearly all of the verbs
> objects, but this also can become a clean way for us to allow drivers
> to export driver-specific objects as well. (eg object_type & (1<<31)
> == driver specific)
> 
> With one more entry point (call_driver_on_object) this would cover a
> lot of cases. Maybe we have many such entry points for performance.
> 
> From there we'd probably have to define a few other common up calls.
> 
> Study of objects:
>  'device','port' - Pre-existing and read-only, query_object returns
>                    information like we have today
>  'pd','mr','mw','cq','srq','qp','ah' - Basic objects, not all have a
> modify/query
>  'flow','xrc',etc - extra objects
> 
> Additional entry points:
>  poll_cq/req_notify_cq - These are high speed, so simple ioctls, or
>                          driver specific
>  post recv/send - drivers implement these via call_driver_on_object
>  attach/detach mcast - Could be 'modify' a qp, or could be new ioctls
>  async_event - Should be an ioctl
>  get_fd - Convert the object to a fd (eg async_fd, comp_channel, etc)
> 
> I think this would substantially address the concern that the uapi is
> 'verbs' or 'qp' specific. Clearly there is space here for any number
> of object families related to RDMA devices.
> 
> It is also discoverable since we can have a query_object that returns
> all 'object_types' the device driver supports.
> 
> Sean, is this more agreeable to you than the unstructured fd idea?

I need to give this more thought, but the general outline seems reasonable.  And I like the generic structure of the ioctl's.

The architecture I was envisioning was something along the lines of having an "rdma_uabi" module.  (And I honestly don't care what things are called; we're abusing names all over.  I'm avoiding the term uverbs because the functionality and structure is different.)  Drivers would plug _directly_ into that module.  This is separate from whatever kernel interfaces that the driver plugs into.  IOCTL dispatch would then go directly to the drivers.  A driver could set its ioctl dispatch to some common kernel function, call core functionality, or just implement what it needs.

Basically, I'm viewing the rdma_uabi as a generic mechanism to export hardware specific interfaces -- be it queue pairs, command queues, or whatever, versus exporting a kernel software interface, like umad or rdma_ucm do.  For better or worse, each hardware driver could independently export their device capabilities.

- Sean