<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [openib-general] Looking for a Mellanox VAPI example C/C++ code</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Hi,</FONT>
</P>
<P><FONT SIZE=2>I don't think we need an example of VAPI code since gen2 will be the real thing.</FONT>
<BR><FONT SIZE=2>When user level verbs will work we will port Mellanox performance test (known as perf_main) to gen2. </FONT>
<BR><FONT SIZE=2>From this test one can learn how to get the good performance out of IB.</FONT>
</P>
<P><FONT SIZE=2>If you have Mellanox SW you can get perf_main sources.</FONT>
<BR><FONT SIZE=2>If you want I can send this code to you, or put it in gen1 in some place (but Makefiles will not work as is)</FONT>
</P>
<P><FONT SIZE=2>Tziporet</FONT>
</P>
<BR>
<P><FONT SIZE=2>-----Original Message-----</FONT>
<BR><FONT SIZE=2>From: Sinate [<A HREF="mailto:3d@sinate.com">mailto:3d@sinate.com</A>]</FONT>
<BR><FONT SIZE=2>Sent: Sunday, February 13, 2005 11:40 PM</FONT>
<BR><FONT SIZE=2>To: Mike Houston</FONT>
<BR><FONT SIZE=2>Cc: openib-general@openib.org</FONT>
<BR><FONT SIZE=2>Subject: Re: [openib-general] Looking for a Mellanox VAPI example C/C++</FONT>
<BR><FONT SIZE=2>code</FONT>
</P>
<BR>
<P><FONT SIZE=2>Thanks Mike. I'll put your views to my boss who is asking me to write </FONT>
<BR><FONT SIZE=2>the VAPI app. So far I've also used SDP, with some pretty reasonable </FONT>
<BR><FONT SIZE=2>results. However, the customer always wants more. I've also put this </FONT>
<BR><FONT SIZE=2>question to Infinicon. I won't be surprised if I don't get a response.</FONT>
</P>
<P><FONT SIZE=2>Guess I better start searching OpenIB archive...</FONT>
</P>
<P><FONT SIZE=2>Cheers,</FONT>
</P>
<P><FONT SIZE=2>Sinate.</FONT>
</P>
<P><FONT SIZE=2>Mike Houston wrote:</FONT>
</P>
<P><FONT SIZE=2>> This has been a running request now for quite some time. Everyone </FONT>
<BR><FONT SIZE=2>> says its a good idea, but it's hard to get it actually done. (And </FONT>
<BR><FONT SIZE=2>> before people say, "this is an open source project so why don't you do </FONT>
<BR><FONT SIZE=2>> it yourself", keep reading) Since the spec is slowly evolving, we </FONT>
<BR><FONT SIZE=2>> really need the OpenIB developers or the IBTA to write a programmer's </FONT>
<BR><FONT SIZE=2>> manual for VAPI. Folks working on the ULPs should do the same.</FONT>
<BR><FONT SIZE=2>> I learned (poorly) how to use VAPI by reading the original Mellanox </FONT>
<BR><FONT SIZE=2>> perf_main code, which was a truly awful experience. This has lead to </FONT>
<BR><FONT SIZE=2>> me writing a VERY fragile NAL using VAPI that has performance </FONT>
<BR><FONT SIZE=2>> characteristics which are very difficult to reason about.</FONT>
<BR><FONT SIZE=2>> Without updated API documentation and programming guides, IB is going </FONT>
<BR><FONT SIZE=2>> to be a hard sell to many folks unless their important apps already </FONT>
<BR><FONT SIZE=2>> work perfectly an fast on IB somehow. For example, what fields do I </FONT>
<BR><FONT SIZE=2>> set in the many different structs to get good performance? The event </FONT>
<BR><FONT SIZE=2>> handlers are cool, but how do you actually use them? How do I figure </FONT>
<BR><FONT SIZE=2>> out if a cool feature is actually supported, i.e. caps bits? We need </FONT>
<BR><FONT SIZE=2>> something like a "HelloIB" for all of the different ways to access the </FONT>
<BR><FONT SIZE=2>> IB hardware (*DAPL, VAPI, SDP (with all the socket extensions, etc)), </FONT>
<BR><FONT SIZE=2>> along with a programmer's guide. Something like if you had to teach </FONT>
<BR><FONT SIZE=2>> an intro undergrad course on networking, but instead of using TCP, you </FONT>
<BR><FONT SIZE=2>> had to teach using IB. What would your slides look like? What would </FONT>
<BR><FONT SIZE=2>> the homework/project questions and solutions be... i.e UD vs. RC. </FONT>
<BR><FONT SIZE=2>> send/recv vs. RDMA (read/write). Basic explanations of </FONT>
<BR><FONT SIZE=2>> WQEs/SRQs/CQs/QPs/etc.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> If you look at the OpenIB archive of this list, you can find an </FONT>
<BR><FONT SIZE=2>> example or two of how to write a ping test. But, getting good </FONT>
<BR><FONT SIZE=2>> performance through VAPI seems to still be a dark art. It's obviously </FONT>
<BR><FONT SIZE=2>> doable since the OSU folks are getting pretty good performance with </FONT>
<BR><FONT SIZE=2>> their MVAPICH MPI implementation on IB. I can get pretty good </FONT>
<BR><FONT SIZE=2>> performance point to point, but 1->N, N->1, M->N, and all to all </FONT>
<BR><FONT SIZE=2>> continue to be a problem for me.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> This is why I've made heavy use out of SDP, but to get all the </FONT>
<BR><FONT SIZE=2>> performance of IB, especially on PCIe, it looks like we will have to </FONT>
<BR><FONT SIZE=2>> write code pretty close to the metal, be it VAPI or kDAPL(?).</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> This is just my opinion as and end user, I could be wrong.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> -Mike</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Sinate wrote:</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> Hi. I'm trying to find a simple Mellanox VAPI example (sends some </FONT>
<BR><FONT SIZE=2>>> data from one host to another, nothing fancy), in C or C++.</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> I can stare at the Mellanox VAPI reference files, but it doesn't get </FONT>
<BR><FONT SIZE=2>>> me any where if I want to build a C/C++ program to do something.</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> Sorry if this is the wrong place to ask.</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> Cheers.</FONT>
<BR><FONT SIZE=2>>> _______________________________________________</FONT>
<BR><FONT SIZE=2>>> openib-general mailing list</FONT>
<BR><FONT SIZE=2>>> openib-general@openib.org</FONT>
<BR><FONT SIZE=2>>> <A HREF="http://openib.org/mailman/listinfo/openib-general" TARGET="_blank">http://openib.org/mailman/listinfo/openib-general</A></FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> To unsubscribe, please visit </FONT>
<BR><FONT SIZE=2>>> <A HREF="http://openib.org/mailman/listinfo/openib-general" TARGET="_blank">http://openib.org/mailman/listinfo/openib-general</A></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
</P>
<P><FONT SIZE=2>_______________________________________________</FONT>
<BR><FONT SIZE=2>openib-general mailing list</FONT>
<BR><FONT SIZE=2>openib-general@openib.org</FONT>
<BR><FONT SIZE=2><A HREF="http://openib.org/mailman/listinfo/openib-general" TARGET="_blank">http://openib.org/mailman/listinfo/openib-general</A></FONT>
</P>
<P><FONT SIZE=2>To unsubscribe, please visit <A HREF="http://openib.org/mailman/listinfo/openib-general" TARGET="_blank">http://openib.org/mailman/listinfo/openib-general</A></FONT>
</P>
</BODY>
</HTML>