[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Paul Grun]

Thanks Christoph.  I know just what you mean about fragmented ideas, so I encourage the email discussion as a means of bringing some cohesion.  Then we can use next Tuesday's meeting to bring the group up to speed and help add some shape to it to see if we can force a proposal to emerge.
-Paul

From: christoph at graphe.net [mailto:christoph at graphe.net] On Behalf Of Christoph Lameter
Sent: Thursday, December 05, 2013 12:40 PM
To: Paul Grun
Cc: 'openframeworkwg at lists.openfabrics.org' (openframeworkwg at lists.openfabrics.org)
Subject: Re: [Openframeworkwg] Some concerns about the new Fabric/IBverbs API

Well my ideas are a bit fragmented right now and I am not sure that I can even call this a proposal (more vague ideas) but I can try to get some more cohesive thoughts together by Tuesday. I have to confess that I am a bit more comfortable with developing ideas via email since that is the traditional mode of operation among kernel developers.
I think we need to explore way to simplify the use of RDMA and at the same time make it more universally applicable. Using established mechanisms like the library system and existing raw device interfaces could be a viable approach.
I just looked at the inline code in ibverbs and it looks like this is simply a move of the indirect function call into the app instead of being in separate function. That is not really what I would call inline. A true inline call would be able to hand the device i/o without any additional function calls. With libraries true inline calls would be possible if one would compile against a specific vendor library. A vendor could provide an inline function f.e. to poll the cq by reading/writing memory mapped location directly from within the app.
If one has done so then the application would be only able to work with a certain device since device specific inline code was generated in the executable.
If we just want the indirection inline then that can also be done with a library approach by setting up vectors there. dlopen and friends can then be used to populate that table and the same approach that ibverbs does today would still work.
How much of a benefit is the current inline indirection implementation in ibverbs.h for ib_post_send etc?

On Thu, Dec 5, 2013 at 1:15 PM, Paul Grun <grun at cray.com<mailto:grun at cray.com>> wrote:
I am replying to Christoph's initial message because it seems to frame the question most clearly.

Christoph, could I persuade you to present your thoughts in the form of a brief proposal?  That might be a good way to frame the question in order to bring the whole up to speed on your idea as well as accelerate the process of coming to closure.  It need not be a detailed proposal, but simply a way to both fully understand the idea and understand its merits.

Correct me if I am wrong, but I think you are proposing two separate, but interrelated ideas:

1.       Leverage existing Linux shared library mechanisms as a "pseudo-framework" for providing application access to provider services

2.       The use of the VFIO interface to all direct memory access transfers (including direct access to low level driver controls).

Is this something you would be willing to take on for next Tuesday's meeting?
Thanks,
-Paul

From: openframeworkwg-bounces at lists.openfabrics.org<mailto:openframeworkwg-bounces at lists.openfabrics.org> [mailto:openframeworkwg-bounces at lists.openfabrics.org<mailto:openframeworkwg-bounces at lists.openfabrics.org>] On Behalf Of Christoph Lameter
Sent: Wednesday, December 04, 2013 8:59 AM
To: 'openframeworkwg at lists.openfabrics.org<mailto:openframeworkwg at lists.openfabrics.org>' (openframeworkwg at lists.openfabrics.org<mailto:openframeworkwg at lists.openfabrics.org>)
Subject: [Openframeworkwg] Some concerns about the new Fabric/IBverbs API

One of the things that worries me is that we develop specialized APIs based on function vectors.
That kind functionality is already provided by the shared library systems in Linux. Could not be each of the modules that we talked about just be mapped to a shared library and define the API simply that way? There are mechanisms in place to deal with compatibility issues.

What are the specific requirements that prevent us from using the standard library approach? If at all possible we should simply use what is there.

Then regarding the separation between user space and kernel space: There are alternate approaches possible to the use of IBverbs. The Linux kernel has added a VFIO interface that allows exposure of device driver controls to user space. If we could base libfabrics on that API instead then the RDMA approach would be generically be usable with any device that supports VFIO.

http://lwn.net/Articles/509153/
We could slim down the components needed in the kernel and provide some simpler management APIs. The IBverbs code per se could be in userspace.
In fact in our company we have some tentative ideas that we could avoid going through IBverbs by going through VFIO. The problem there is that no standard user space APIs exist to interact with a device. Having such an API would allow a generic way to use RDMA with devices. We have a couple o vendors that are already providing some simplistic proprietary libraries here. Standardization would be good.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20131205/5984f7f6/attachment.html>