[ofw] Which API to use for IB project

[Tzachi Dar]

I guess that if you can live with latency of ~5us you should be using SDP which will give you the minimum development effort.

If you want to be in the area of ~1us latency you should be using one of the following 3 options IBAL ND or winverbs. I guess that once it comes to latency this are the best choices.

The difference between the other 3 are relatively small, and are more in the way the API is used, support for other libraries and operating systems supported.

IBAL is the oldest of the 3, supported on windows XP 32 and up. It is the most stable of all and programming it is nothing like Linux.
ND and winverbs tend to replace IBAL one day (at least on user mode).


ND officially exists on 2008 and is the official MS RDMA api (I believe it can be used also on 2003). This is the interface that is used by MS-MPI and it is relatively stable. It is layered on either IBAL or winverbs.

Winverbs is a new effort that is being done to supply verbs. It is still under development but going forward it should give all services and replace IBAL. I'm not sure that it is supported on XP 32 but it should be starting on windows server 2003 and up. Since it seats bellow ND it should give somewhat better performance. It's verbs are also relatively close to the Linux verbs.

Please note that to get the best latency one should be using RDMA write and pool on the memory for completion (see the ib_write_lat program for an example).

Thanks
Tzachi

> -----Original Message-----
> From: Thomas Peiselt [mailto:dispanser at googlemail.com] 
> Sent: Monday, August 03, 2009 3:35 PM
> To: Tzachi Dar
> Cc: ofw at lists.openfabrics.org
> Subject: Re: [ofw] Which API to use for IB project
> 
> Hi,
> 
> The application is in user mode.
> 
> Thomas
> 
> On Mon, Aug 3, 2009 at 2:24 PM, Tzachi 
> Dar<tzachid at mellanox.co.il> wrote:
> > Is your application on user mode or kernel mode?
> >
> > Thanks
> > Tzachi
> >
> >> -----Original Message-----
> >> From: ofw-bounces at lists.openfabrics.org 
> >> [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Thomas 
> >> Peiselt
> >> Sent: Monday, August 03, 2009 3:22 PM
> >> To: ofw at lists.openfabrics.org
> >> Subject: [ofw] Which API to use for IB project
> >>
> >> Hello,
> >>
> >> I'm new to InfiniBand (and this list), and I'm not sure if 
> the topic 
> >> is adequate on this list. If not, don't kill me, show me the right 
> >> way instead :-).
> >>
> >> My task: given two shiny new workstations w/ Mellanox IB HCAs (the 
> >> cheap InfniHost  Lx cards), adapt the companies internal 
> >> communication library to work over IB - as seamless as 
> possible, as 
> >> fast as possible... um yeah, you know, I guess .
> >>
> >> I am lost in what a InfiniBand newbie might consider an 
> API / library
> >> jungle: there's ND, WSD, SDP, IPoIB, uDAPL, and WinVerbs. I'm not 
> >> sure where to start. I understand that some APIs take zero 
> or almost 
> >> zero porting effort (IPoIB, WSD), but do not seem to take full 
> >> advantage of all the IB features (zero copy, cpu-bypass).
> >>
> >> My question: what approach to chose for an application 
> where packet 
> >> size is relatively small (less than 1024 bytes) with the following
> >> priorities:
> >> 1. latency
> >> 2. cpu load
> >> 3. low porting effort to linux compute nodes 4. 
> implementation effort
> >>
> >> Latency is crucial, porting / implementation effort is 
> just nice to 
> >> have.
> >>
> >> thanks for any suggestions,
> >>
> >> Thomas
> >> _______________________________________________
> >> ofw mailing list
> >> ofw at lists.openfabrics.org
> >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
> >>
> >
>