<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 4, 2013 at 11:53 AM, Hefty, Sean <span dir="ltr"><<a href="mailto:sean.hefty@intel.com" target="_blank">sean.hefty@intel.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">> That kind functionality is already provided by the shared library systems<br>
> in Linux. Could not be each of the modules that we talked about just be<br>
> mapped to a shared library and define the API simply that way? There are<br>
> mechanisms in place to deal with compatibility issues.<br>
<br>
</div>These APIs are usable together, not stand-alone. The calls used to report events are used with the data transfer calls. And different data transfer calls work with the same endpoint (e.g. message send and RDMA write).</blockquote>
<div><br></div><div>The same can be said done using libraries if you create the proper dependencies. </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">
> What are the specific requirements that prevent us from using the standard<br>
> library approach? If at all possible we should simply use what is there.<br>
<br>
</div>Can you provide more specific details on how you see this working? I'm not following what sort of changes or approach you're thinking about.<br></blockquote><div><br></div><div>I am not clear on why we have this complicated API with the extension when the linker etc already provide such a framework with associated tools to manage them. There is a history for example of adding/modifying functions to glibc as well to the kernel. Do we really cannot use the established means of defining user space APIs?<br>
</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This is what we have with libibverbs today:<br>
<br>
libibverbs exports a set of function calls, such as<br>
<br>
struct ibv_pd *ibv_alloc_pd(struct ibv_context *context);<br>
struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr, size_t length, int access);<br>
struct ibv_cq *ibv_create_cq(struct ibv_context *context, int cqe,<br>
void *cq_context, ibv_comp_channel *channel, comp_vector);<br>
struct ibv_qp *ibv_create_qp(struct ibv_pd *pd, ibv_qp_init_attr *qp_init_attr);<br>
int ibv_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, attr_mask);<br>
...<br>
<br>
libibverbs itself can't implement these calls, since the implementation is dependent on the provider. So the provider plugs a set of function calls into libibverbs.<br></blockquote><div><br></div><div>Isnt the loader used to load an provider implementation that can then be used by ibverbs? Can these not be simple functions provided by an provider provided library instead of a vector of functions? dlopen and friends can manage that.</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
struct ibv_context_ops {<br>
int (*query_device)(struct ibv_context *context,<br>
struct ibv_device_attr *device_attr);<br>
int (*query_port)(struct ibv_context *context, uint8_t port_num,<br>
struct ibv_port_attr *port_attr);<br>
struct ibv_pd * (*alloc_pd)(struct ibv_context *context);<br>
int (*dealloc_pd)(struct ibv_pd *pd);<br>
struct ibv_mr * (*reg_mr)(struct ibv_pd *pd, void *addr, size_t length,<br>
int access);<br>
...<br>
<br>
libibverbs can then map calls to the provider, in some cases using static inline function. Whether a static inline call is used or not, the basic operation is the same.<br></blockquote><div><br></div><div>The static inline could be an issue. Not sure how to do that.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
static inline int ibv_poll_cq(struct ibv_cq *cq, int num_entries, struct ibv_wc *wc)<br>
{<br>
return cq->context->ops.poll_cq(cq, num_entries, wc);<br>
}<br></blockquote><div><br></div><div>Well dlopen can be used to threat libraries as essentially a vector of functions if you wanted to to do that. So providers could write an implementation providing a prescribed set of functions that would then be converted to a vector array if we need the above.</div>
<div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
>From one viewpoint, I'm suggesting replacing 'struct ibv_context_ops' with 'struct ibv_qp_ops' + 'struct ibv_cq_ops' + 'struct ibv_mr_ops' + ..., and using static inline functions similar to ibv_poll_cq().<br>
<br>
I don't see how we can get away from using function pointers into the providers.</blockquote><div><br></div><div>If you must have these function pointers for certain use cases then the existing library mechanism can also provide that. Would not function definitions be better for the rest because we can then avoid the indirection? Plus we would have a rich set of tools to manage the APIs.</div>
<div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
> Then regarding the separation between user space and kernel space: There<br>
> are alternate approaches possible to the use of IBverbs. The Linux kernel<br>
> has added a VFIO interface that allows exposure of device driver controls<br>
> to user space. If we could base libfabrics on that API instead then the<br>
> RDMA approach would be generically be usable with any device that supports<br>
> VFIO.<br>
><br>
> <a href="http://lwn.net/Articles/509153/" target="_blank">http://lwn.net/Articles/509153/</a><br>
<br>
</div>Are you suggesting using VFIO as the interface between user space and the kernel, rather than dropping through the kernel ib_user_verbs module?<br>
<br>
Do you see the use of VFIO impacting the user space APIs or the internal architecture of the framework or both?</blockquote><div><br></div><div>VFIO could be used as a base API for getting access to the device. Then we may need to have some additional syscalls that may be required to provide functionality that the existing VFIO interface does not provide.</div>
<div><br></div><div>Vendors are already able to write an user space driver just based on the VFIO spec. If the device supports VFIO then basically also RDMA is possible by acquiring a VFIO instance of the device and managing the RX and TX queues in user space. The mellanox driver supports VFIO f.e.</div>
<div><br></div><div>One problem may be that one VFIO instance is needed per process that uses offload/RDMA and also per device. For our needs however that means that we have full control of the device at the register level. Which seems desirable to many of us and would allow the full use of all the features of the hardware. The API is vendor specific though. What would be needed is a set of higher level libraries that allow the vendor to provide a userspace library that then ties into the RDMA framework to make the use of these devices easy from userspace.</div>
<div><br></div><div><br></div><div><br></div><div><br></div></div><br><br></div></div>