[ofa-general] Memory registration limit of 16GB with Chelsio

[Felix Marti]

-----Original Message-----
From: Steve Wise [mailto:swise at opengridcomputing.com] 
Sent: Friday, July 10, 2009 11:46 AM
To: pandit ib
Cc: general at lists.openfabrics.org; Felix Marti
Subject: Re: [ofa-general] Memory registration limit of 16GB with
Chelsio

Looks like you're running out of adapter resources for storing the 
physical buffers lists (PBLs).

The cards have 32MB of memory for storing PBLs.

32MB of memory can store 4MB of 8B physical addresses.

For user mode, your page size is 4KB, so that maps out to 4MB * 4KB =  
16GB of memory.

Hey Felix,

Can we somehow adjust the PMRX memory map dynamically?  For instance, 
maybe drop the number of TPTs to allow for more PBL memory?
[felix] Not dynamically, but it can surely be changed. DM, can it be
controlled through a cxgbtool?


Steve

pandit ib wrote:
> Hi,
>
> In our testing we are hitting a memory registration limit of 16GB with
> the Chelsio cards.
>
> Here are some sample results for different region sizes.
>
> region size                 Number of Successful registration
Total
> 2MB                          8178
>              ~16GB
> 16MB                        1021
>             ~16GB
> 128MB                       127
>             ~16GB
>
> Are there any knobs that we can use to raise that limit?
>
> For example, on mlx4 we use "log_num_mtt" module load parameter to
> increase the limit.
>
> Here is a sample output of the test:
>
> [root at lv2 ~]# ./mr-test 16777216
>
> mr-test: bufsize 16777216
> device # 0 name="cxgb3_0" guid="00074305ca890000"
> 	ibv_open_device() context=0x77b5100
> 	ibv_alloc_pd() pd=0x77b50e0
> 	alloc: 1021
> 	ibv_reg_mr failed:: Cannot allocate memory
> 	fw_ver: 7.4.0
> 	max_mr_size 0x100000000
> 	max_mr: 32768, could only register 1021 regions; Region size
16MB
> 	sleep 5 sec
> 	free: 0
> done
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
>