[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Paul Grun]

I am replying to Christoph's initial message because it seems to frame the question most clearly.

Christoph, could I persuade you to present your thoughts in the form of a brief proposal?  That might be a good way to frame the question in order to bring the whole up to speed on your idea as well as accelerate the process of coming to closure.  It need not be a detailed proposal, but simply a way to both fully understand the idea and understand its merits.

Correct me if I am wrong, but I think you are proposing two separate, but interrelated ideas:

1.       Leverage existing Linux shared library mechanisms as a "pseudo-framework" for providing application access to provider services

2.       The use of the VFIO interface to all direct memory access transfers (including direct access to low level driver controls).

Is this something you would be willing to take on for next Tuesday's meeting?
Thanks,
-Paul

From: openframeworkwg-bounces at lists.openfabrics.org [mailto:openframeworkwg-bounces at lists.openfabrics.org] On Behalf Of Christoph Lameter
Sent: Wednesday, December 04, 2013 8:59 AM
To: 'openframeworkwg at lists.openfabrics.org' (openframeworkwg at lists.openfabrics.org)
Subject: [Openframeworkwg] Some concerns about the new Fabric/IBverbs API

One of the things that worries me is that we develop specialized APIs based on function vectors.
That kind functionality is already provided by the shared library systems in Linux. Could not be each of the modules that we talked about just be mapped to a shared library and define the API simply that way? There are mechanisms in place to deal with compatibility issues.

What are the specific requirements that prevent us from using the standard library approach? If at all possible we should simply use what is there.

Then regarding the separation between user space and kernel space: There are alternate approaches possible to the use of IBverbs. The Linux kernel has added a VFIO interface that allows exposure of device driver controls to user space. If we could base libfabrics on that API instead then the RDMA approach would be generically be usable with any device that supports VFIO.

http://lwn.net/Articles/509153/
We could slim down the components needed in the kernel and provide some simpler management APIs. The IBverbs code per se could be in userspace.
In fact in our company we have some tentative ideas that we could avoid going through IBverbs by going through VFIO. The problem there is that no standard user space APIs exist to interact with a device. Having such an API would allow a generic way to use RDMA with devices. We have a couple o vendors that are already providing some simplistic proprietary libraries here. Standardization would be good.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20131205/9ae8422d/attachment.html>