[ofa-general] Memory registration limit of 16GB with Chelsio

[pandit ib]

Hi,

In our testing we are hitting a memory registration limit of 16GB with
the Chelsio cards.

Here are some sample results for different region sizes.

region size                 Number of Successful registration        Total
2MB                          8178
             ~16GB
16MB                        1021
            ~16GB
128MB                       127
            ~16GB

Are there any knobs that we can use to raise that limit?

For example, on mlx4 we use "log_num_mtt" module load parameter to
increase the limit.

Here is a sample output of the test:

[root at lv2 ~]# ./mr-test 16777216

mr-test: bufsize 16777216
device # 0 name="cxgb3_0" guid="00074305ca890000"
	ibv_open_device() context=0x77b5100
	ibv_alloc_pd() pd=0x77b50e0
	alloc: 1021
	ibv_reg_mr failed:: Cannot allocate memory
	fw_ver: 7.4.0
	max_mr_size 0x100000000
	max_mr: 32768, could only register 1021 regions; Region size 16MB
	sleep 5 sec
	free: 0
done