[openib-general] [PATCH] FMR support in mthca

[Libor Michalek]

On Tue, Mar 29, 2005 at 05:08:14PM -0800, Grant Grundler wrote:
> On Mon, Mar 28, 2005 at 05:03:51PM -0800, Libor Michalek wrote:
> >   I haven't looked closely at the code yet, but I did try it out
> > with SDP/AIO on a pair of x86 systems with Tavors and a pair of 
> > x86_64 systems with Arbels. With a small change to core/fmr_pool.c 
> > and enabling pool creation in SDP it worked as expected. Here are 
> > throughput results:
> > 
> >                      x86       x86_64
> >                     --------   --------
> >   SDP  sync         610 MB/s   710 MB/s
> >   SDP async (hit)   740 MB/s   910 MB/s
> >   SDP async (miss)  590 MB/s   910 MB/s
> 
> Libor,
> How did you generate the above numbers?  netpipe?
> 
> I'd like to add the "HP ZX1" (and maybe parisc) column.

  I used ttcp which was recompiled to use the SDP protocol family,
and a modified ttcp for the async numbers. The modified ttcp replaced
the socket send/recv system calls with socket AIO io_submit/io_getevents
system calls.

  The recompile modifications to use SDP protocol familt for regular 
ttcp are pretty straight forward:

  #include <sdp_sock.h>
  #undef AF_INET
  #define AF_INET AF_INET_SDP

  I could checkin the source for the modified ttcp's but I'm not sure
exactly where... (gen2/users/libor ???)

> BTW, I've got several systems loaded with:
> 
> and zero clue how to get it to talk to another system.
> ie "ifconfig -a" isn't listing any new interfaces. :^(

  That's pretty much all you need, once ipoib is configure a socket
created with the SDP protocol family can connect/bind using the
addresses of the ipoib interfaces as you would for a TCP socket.

> The most recent "How-To test SDP" for gen1 posted on Jun 17, 2004:
> 	http://openib.org/pipermail/openib-general/2004-June/002892.html
> 
> It references libsdp.so which doesn't seem to exist in gen2.
> Has anyone written an update for gen2?

  MST checked libsdp into the gen2 tree: gen2/trunk/src/userspace/libsdp
Which is great for using SDP with application binaries that you do not
want to modify. However, for the async numbers you need a program that's
using AIO for network sockets, of which I have a few...


-Libor