[ofa-general] Memory registration limit of 16GB with Chelsio
Steve Wise
swise at opengridcomputing.com
Fri Jul 10 11:45:33 PDT 2009
Looks like you're running out of adapter resources for storing the
physical buffers lists (PBLs).
The cards have 32MB of memory for storing PBLs.
32MB of memory can store 4MB of 8B physical addresses.
For user mode, your page size is 4KB, so that maps out to 4MB * 4KB =
16GB of memory.
Hey Felix,
Can we somehow adjust the PMRX memory map dynamically? For instance,
maybe drop the number of TPTs to allow for more PBL memory?
Steve
pandit ib wrote:
> Hi,
>
> In our testing we are hitting a memory registration limit of 16GB with
> the Chelsio cards.
>
> Here are some sample results for different region sizes.
>
> region size Number of Successful registration Total
> 2MB 8178
> ~16GB
> 16MB 1021
> ~16GB
> 128MB 127
> ~16GB
>
> Are there any knobs that we can use to raise that limit?
>
> For example, on mlx4 we use "log_num_mtt" module load parameter to
> increase the limit.
>
> Here is a sample output of the test:
>
> [root at lv2 ~]# ./mr-test 16777216
>
> mr-test: bufsize 16777216
> device # 0 name="cxgb3_0" guid="00074305ca890000"
> ibv_open_device() context=0x77b5100
> ibv_alloc_pd() pd=0x77b50e0
> alloc: 1021
> ibv_reg_mr failed:: Cannot allocate memory
> fw_ver: 7.4.0
> max_mr_size 0x100000000
> max_mr: 32768, could only register 1021 regions; Region size 16MB
> sleep 5 sec
> free: 0
> done
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
More information about the general
mailing list