[ofa-general] [RFC/PATCH 2/2] RDMA/cxgb3: Fix severe limit on userspace memory registration size

[Steve Wise]

Roland Dreier wrote:
> Currently, iw_cxgb3 is severely limited on the amount of userspace
> memory that can be registered in in a single memory region, which
> causes big problems for applications that expect to be able to
> register 100s of MB.
>
> The problem is that the driver uses a single kmalloc()ed buffer to
> hold the physical buffer list (PBL) for the entire memory region
> during registration, which means that 8 bytes of contiguous memory are
> required for each page of memory being registered.  For example, a 64
> MB registration will require 128 KB of contiguous memory with 4 KB
> pages, and it unlikely that such an allocation will succeed on a busy
> system.
>
> This is purely a driver problem: the temporary page list buffer is not
> needed by the hardware, so we can fix this by writing the PBL to the
> hardware in page-sized chunks rather than all at once.  We do this by
> splitting the memory registration operation up into several steps:
>
>  - Allocate PBL space in adapter memory for the full registration
>  - Copy PBL to adapter memory in chunks
>  - Allocate STag and enable memory region
>
> This also allows several other cleanups to the __cxio_tpt_op()
> interface and related parts of the driver.
>
> This change leaves the reregister memory region and memory window
> operations broken, but they already didn't work due to other
> longstanding bugs, so fixing them will be left to a later patch.
>
> Signed-off-by: Roland Dreier <rolandd at cisco.com>
>   
Acked-by: Steve Wise <swise at opengridcomputing.com>