[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Christoph Lameter]

Well my ideas are a bit fragmented right now and I am not sure that I can
even call this a proposal (more vague ideas) but I can try to get some more
cohesive thoughts together by Tuesday. I have to confess that I am a bit
more comfortable with developing ideas via email since that is the
traditional mode of operation among kernel developers.

I think we need to explore way to simplify the use of RDMA and at the same
time make it more universally applicable. Using established mechanisms like
the library system and existing raw device interfaces could be a viable
approach.

I just looked at the inline code in ibverbs and it looks like this is
simply a move of the indirect function call into the app instead of being
in separate function. That is not really what I would call inline. A true
inline call would be able to hand the device i/o without any additional
function calls. With libraries true inline calls would be possible if one
would compile against a specific vendor library. A vendor could provide an
inline function f.e. to poll the cq by reading/writing memory mapped
location directly from within the app.

If one has done so then the application would be only able to work with a
certain device since device specific inline code was generated in the
executable.

If we just want the indirection inline then that can also be done with a
library approach by setting up vectors there. dlopen and friends can then
be used to populate that table and the same approach that ibverbs does
today would still work.

How much of a benefit is the current inline indirection implementation in
ibverbs.h for ib_post_send etc?


On Thu, Dec 5, 2013 at 1:15 PM, Paul Grun <grun at cray.com> wrote:

>  I am replying to Christoph’s initial message because it seems to frame
> the question most clearly.
>
>
>
> Christoph, could I persuade you to present your thoughts in the form of a
> brief proposal?  That might be a good way to frame the question in order to
> bring the whole up to speed on your idea as well as accelerate the process
> of coming to closure.  It need not be a detailed proposal, but simply a way
> to both fully understand the idea and understand its merits.
>
>
>
> Correct me if I am wrong, but I think you are proposing two separate, but
> interrelated ideas:
>
> 1.       Leverage existing Linux shared library mechanisms as a
> “pseudo-framework” for providing application access to provider services
>
> 2.       The use of the VFIO interface to all direct memory access
> transfers (including direct access to low level driver controls).
>
>
>
> Is this something you would be willing to take on for next Tuesday’s
> meeting?
>
> Thanks,
>
> -Paul
>
>
>
> *From:* openframeworkwg-bounces at lists.openfabrics.org [mailto:
> openframeworkwg-bounces at lists.openfabrics.org] *On Behalf Of *Christoph
> Lameter
> *Sent:* Wednesday, December 04, 2013 8:59 AM
> *To:* 'openframeworkwg at lists.openfabrics.org' (
> openframeworkwg at lists.openfabrics.org)
> *Subject:* [Openframeworkwg] Some concerns about the new Fabric/IBverbs
> API
>
>
>
> One of the things that worries me is that we develop specialized APIs
> based on function vectors.
>
> That kind functionality is already provided by the shared library systems
> in Linux. Could not be each of the modules that we talked about just be
> mapped to a shared library and define the API simply that way? There are
> mechanisms in place to deal with compatibility issues.
>
> What are the specific requirements that prevent us from using the standard
> library approach? If at all possible we should simply use what is there.
>
>
>
> Then regarding the separation between user space and kernel space: There
> are alternate approaches possible to the use of IBverbs. The Linux kernel
> has added a VFIO interface that allows exposure of device driver controls
> to user space. If we could base libfabrics on that API instead then the
> RDMA approach would be generically be usable with any device that supports
> VFIO.
>
> http://lwn.net/Articles/509153/
>
> We could slim down the components needed in the kernel and provide some
> simpler management APIs. The IBverbs code per se could be in userspace.
>
> In fact in our company we have some tentative ideas that we could avoid
> going through IBverbs by going through VFIO. The problem there is that no
> standard user space APIs exist to interact with a device. Having such an
> API would allow a generic way to use RDMA with devices. We have a couple o
> vendors that are already providing some simplistic proprietary libraries
> here. Standardization would be good.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20131205/cacc89d3/attachment.html>