[openib-general] Looking for a Mellanox VAPI example C/C++ co de

[Mike Houston]

Tziporet Koren wrote:

> Hi,
>
> I don't think we need an example of VAPI code since gen2 will be the 
> real thing.
> When user level verbs will work we will port Mellanox performance test 
> (known as perf_main) to gen2.
> From this test one can learn how to get the good performance out of IB.
>
NO, NO, NO.  If you expect people to learn how to use IB (well) from 
code like perf_main, I think you are very mistaken.  User level verbs 
would be fine, if the performance is there, but we would still need FULL 
documentation and a programmer's manual.  While I'll agree that if the 
ULPs can provide good performance, users might not need to code to VAPI, 
but I'll believe it when I see it.  Regardless, programming 
documentation is still required at all levels .  What are the people 
writing ULPs supposed to use?

> If you have Mellanox SW you can get perf_main sources.
> If you want I can send this code to you, or put it in gen1 in some 
> place (but Makefiles will not work as is)
>
As I said, I have the source to perf_main, that is how I learned to 
write VAPI code poorly...

-Mike

> Tziporet
>
>
> -----Original Message-----
> From: Sinate [mailto:3d at sinate.com]
> Sent: Sunday, February 13, 2005 11:40 PM
> To: Mike Houston
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] Looking for a Mellanox VAPI example C/C++
> code
>
>
> Thanks Mike. I'll put your views to my boss who is asking me to write
> the VAPI app. So far I've also used SDP, with some pretty reasonable
> results. However, the customer always wants more. I've also put this
> question to Infinicon. I won't be surprised if I don't get a response.
>
> Guess I better start searching OpenIB archive...
>
> Cheers,
>
> Sinate.
>
> Mike Houston wrote:
>
> > This has been a running request now for quite some time.  Everyone
> > says its a good idea, but it's hard to get it actually done.  (And
> > before people say, "this is an open source project so why don't you do
> > it yourself", keep reading)  Since the spec is slowly evolving, we
> > really need the OpenIB developers or the IBTA to write a programmer's
> > manual for VAPI.  Folks working on the ULPs should do the same.
> > I learned (poorly) how to use VAPI by reading the original Mellanox
> > perf_main code, which was a truly awful experience.  This has lead to
> > me writing a VERY fragile NAL using VAPI that has performance
> > characteristics which are very difficult to reason about.
> > Without updated API documentation and programming guides, IB is going
> > to be a hard sell to many folks unless their important apps already
> > work perfectly an fast on IB somehow.  For example, what fields do I
> > set in the many different structs to get good performance?  The event
> > handlers are cool, but how do you actually use them?  How do I figure
> > out if a cool feature is actually supported, i.e. caps bits?  We need
> > something like a "HelloIB" for all of the different ways to access the
> > IB hardware (*DAPL, VAPI, SDP (with all the socket extensions, etc)),
> > along with a programmer's guide.  Something like if you had to teach
> > an intro undergrad course on networking, but instead of using TCP, you
> > had to teach using IB.  What would your slides look like?  What would
> > the homework/project questions and solutions be...  i.e UD vs. RC. 
> > send/recv vs. RDMA (read/write).  Basic explanations of
> > WQEs/SRQs/CQs/QPs/etc.
> >
> > If you look at the OpenIB archive of this list, you can find an
> > example or two of how to write a ping test.  But, getting good
> > performance through VAPI seems to still be a dark art.  It's obviously
> > doable since the OSU folks are getting pretty good performance with
> > their MVAPICH MPI implementation on IB.  I can get pretty good
> > performance point to point, but 1->N, N->1, M->N, and all to all
> > continue to be a problem for me.
> >
> > This is why I've made heavy use out of SDP, but to get all the
> > performance of IB, especially on PCIe, it looks like we will have to
> > write code pretty close to the metal, be it VAPI or kDAPL(?).
> >
> > This is just my opinion as and end user, I could be wrong.
> >
> > -Mike
> >
> > Sinate wrote:
> >
> >> Hi. I'm trying to find a simple Mellanox VAPI example (sends some
> >> data from one host to another, nothing fancy), in C or C++.
> >>
> >> I can stare at the Mellanox VAPI reference files, but it doesn't get
> >> me any where if I want to build a C/C++ program to do something.
> >>
> >> Sorry if this is the wrong place to ask.
> >>
> >> Cheers.
> >> _______________________________________________
> >> openib-general mailing list
> >> openib-general at openib.org
> >> http://openib.org/mailman/listinfo/openib-general
> >>
> >> To unsubscribe, please visit
> >> http://openib.org/mailman/listinfo/openib-general
> >
> >
> >
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>