[openib-general] [PATCH] FMR support in mthca

[Libor Michalek]

On Fri, Apr 01, 2005 at 05:56:00PM -0500, Hal Rosenstock wrote:
> On Thu, 2005-03-31 at 16:41, Libor Michalek wrote: 
> > 
> >      ./ttcp.aio.x -r -l 98304 -a 20 -f M
> >      ./ttcp.aio.x -t -l 98304 -n 200000 -a 20 -f M 192.168.0.100
> > 
> > > > For the FMR pool cache miss async results I used 1000 different
> > > > buffers, of which only 20 were in flight at a time.
> > 
> >      ./ttcp.aio.x -r -l 98304 -a 20 -x 1000 -f M
> >      ./ttcp.aio.x -t -l 98304 -n 200000 -a 20 -x 1000 -f M 192.168.0.100
> 
> We are seeing issues with both buffer size and iterations. We get back
> -ENOMEM and also see VMA lock errors. Are the 2 related ? Should we turn
> on SDP debug to see what specifically can't be allocated ? In that case,
> what could be done ?

Hal,

  You need to increase the amount of memory that the user is allowed
to lock. The following command in each shell from which you are running
ttcp:

  limit memorylocked unlimited

  In 2.6 mlock() looks at the rlimits for the process executing the lock,
I decided not to artificially increase the limit while locking, instead
relying on the user/admin to set the appropriate value. The default is
pretty small.

-Libor