[openib-general] [PATCH] FMR support in mthca

[Libor Michalek]

On Wed, Mar 30, 2005 at 10:09:24AM -0500, Hal Rosenstock wrote:
> On Mon, 2005-03-28 at 20:03, Libor Michalek wrote:
> >   I haven't looked closely at the code yet, but I did try it out
> > with SDP/AIO on a pair of x86 systems with Tavors and a pair of 
> > x86_64 systems with Arbels. With a small change to core/fmr_pool.c 
> > and enabling pool creation in SDP it worked as expected. Here are 
> > throughput results:
> > 
> >                      x86       x86_64
> >                     --------   --------
> >   SDP  sync         610 MB/s   710 MB/s
> >   SDP async (hit)   740 MB/s   910 MB/s
> >   SDP async (miss)  590 MB/s   910 MB/s
> > 
> >   For sync sockets I used 81600 byte buffers. For async socket I kept
> > 20 96K buffers in flight. For the FMR pool cache hit async results I 
> > used  only 20 different buffers. For the FMR pool cache miss async
> > results I used 1000 different buffers, of which only 20 were in flight
> > at a time.
> 
> Any idea why hit/miss make such a difference on x86 and not x86_64 ?

  Not sure, but the difference is also Tavor vs. Arbel on those two
pairs of systems.

> Also, is all the code for this now checked available ?

  Almost, for SDP I need to check in this patch, now that the FMR code
has been checked in.

-Libor

Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 2072)
+++ sdp_conn.c	(working copy)
@@ -1774,14 +1774,12 @@
 	/*
 	 * create SDP memory pool
 	 */
-#ifdef _FMR_SUPPORT
 	hca->fmr_pool = ib_create_fmr_pool(hca->pd, &fmr_param_s);
 	if (IS_ERR(hca->fmr_pool)) {
 		sdp_warn("Error <%ld> creating HCA <%s> fast memory pool",
 			 PTR_ERR(hca->fmr_pool), device->name);
 		goto error;
 	}
-#endif
 	/*
 	 * port allocation
 	 */
@@ -1828,10 +1826,9 @@
 		kfree(port);
 	}
 
-#ifdef _FMR_SUPPORT
 	if (!IS_ERR(hca->fmr_pool))
 		(void)ib_destroy_fmr_pool(hca->fmr_pool);
-#endif
+
 	if (!IS_ERR(hca->mem_h))
 		(void)ib_dereg_mr(hca->mem_h);
 
@@ -1864,10 +1861,9 @@
 		kfree(port);
 	}
 
-#ifdef _FMR_SUPPORT
 	if (!IS_ERR(hca->fmr_pool))
 		(void)ib_destroy_fmr_pool(hca->fmr_pool);
-#endif	
+
 	if (!IS_ERR(hca->mem_h))
 		(void)ib_dereg_mr(hca->mem_h);