[ofa-general] [PATCH 0/3] V2 - libmthca libmlx4 - Optimize memory allocation of QP buffers

[sebastien dugue]

  Hi,

  here is a re-spin of the QP buffers memory allocation optimization patches in
which QP buffers are allocated using mmap() regardless of the page size.


Changes V1 -> V2:
----------------

  - Use mmap whatever the page size, not only with 64K pages.


  libmthca and libmlx4 allocate QP buffers using posix_memalign(), which
results in big memory wastage on architectures with 64K pages.

  Replacing posix_memalign() with mmap() allows to fix this (more description
in the patches themselves).

  Now, for some numbers, a micro benchmark I wrote shows the heap usage and
the number of mmaped pages used with posix_memalign() and mmap() respectively
for 1000, 2000, up to 8000 QP.

  MTHCA
               posix_memalign			   mmap
  QP	   heap		mmaped(pages)	   heap		mmaped(pages)
 1000	   838736	    2988	   576512	   1000
 2000	  1751216	    5973	  1161264	   2000
 3000	  2598144	    8961	  1746016	   3000
 4000	  3510656	   11946	  2330704	   4000
 5000	  4357616	   14934	  2915440	   5000
 6000	  5270080	   17919	  3500176	   6000
 7000	  6117056	   20907	  4084912	   7000
 8000	  6963968	   23895	  4669632	   8000

  MLX4
               posix_memalign			   mmap
  QP	   heap		mmaped(pages)	   heap		mmaped(pages)
 1000	  1469424	    2982	  1010544	   1003
 2000	  2994048	    5958	  2010752	   2003
 3000	  4518672	    8934	  3010960	   3003
 4000	  5969520	   11913	  4002960	   4003
 5000	  7494176	   14889	  5003168	   5003
 6000	  8953248	   17868	  6003376	   6003
 7000	 10477856	   20844	  7003584	   7003
 8000	 12002496	   23820	  8003792	   8003


  This patchset consists in 3 patches:

  1. Optimize memory allocation of QP buffers for libmthca
  2. Optimize memory allocation of QP buffers for libmlx4
  3. Fix the fixes patches for libmlx4 after having applied the
     previous patch.

  Sebastien Dugue