[openib-general] Looking for a Mellanox VAPI example C/C++ code

[Mike Houston]

This has been a running request now for quite some time.  Everyone says 
its a good idea, but it's hard to get it actually done.  (And before 
people say, "this is an open source project so why don't you do it 
yourself", keep reading)  Since the spec is slowly evolving, we really 
need the OpenIB developers or the IBTA to write a programmer's manual 
for VAPI.  Folks working on the ULPs should do the same. 

I learned (poorly) how to use VAPI by reading the original Mellanox 
perf_main code, which was a truly awful experience.  This has lead to me 
writing a VERY fragile NAL using VAPI that has performance 
characteristics which are very difficult to reason about. 

Without updated API documentation and programming guides, IB is going to 
be a hard sell to many folks unless their important apps already work 
perfectly an fast on IB somehow.  For example, what fields do I set in 
the many different structs to get good performance?  The event handlers 
are cool, but how do you actually use them?  How do I figure out if a 
cool feature is actually supported, i.e. caps bits?  We need something 
like a "HelloIB" for all of the different ways to access the IB hardware 
(*DAPL, VAPI, SDP (with all the socket extensions, etc)), along with a 
programmer's guide.  Something like if you had to teach an intro 
undergrad course on networking, but instead of using TCP, you had to 
teach using IB.  What would your slides look like?  What would the 
homework/project questions and solutions be...  i.e UD vs. RC.  
send/recv vs. RDMA (read/write).  Basic explanations of 
WQEs/SRQs/CQs/QPs/etc.

If you look at the OpenIB archive of this list, you can find an example 
or two of how to write a ping test.  But, getting good performance 
through VAPI seems to still be a dark art.  It's obviously doable since 
the OSU folks are getting pretty good performance with their MVAPICH MPI 
implementation on IB.  I can get pretty good performance point to point, 
but 1->N, N->1, M->N, and all to all continue to be a problem for me.

This is why I've made heavy use out of SDP, but to get all the 
performance of IB, especially on PCIe, it looks like we will have to 
write code pretty close to the metal, be it VAPI or kDAPL(?).

This is just my opinion as and end user, I could be wrong.

-Mike

Sinate wrote:

> Hi. I'm trying to find a simple Mellanox VAPI example (sends some data 
> from one host to another, nothing fancy), in C or C++.
>
> I can stare at the Mellanox VAPI reference files, but it doesn't get 
> me any where if I want to build a C/C++ program to do something.
>
> Sorry if this is the wrong place to ask.
>
> Cheers.
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general