[ofa-general] max_mr limit

[Jack Morgenstein]

On Monday 17 March 2008 06:49, Rajouri Jammu wrote:
> Jack,
> 
> The problem I'm seeing is that I'm not able to register even the default
> number of memory regions.
> The default is 131056 but I'm not able to register more than 32635 regions
> I'm on OFED -1.2.5.4.
> 
> Any ideas?
> 
You are running out of MTTs.  Roughly speaking, by default
there are 2^17 (= 131K) mpts, and 2^20 mtts.  Each memory region
uses 1 mpt entry, and an MTT entry for each 4K of the region.
Thus, to be able to allocate the maximum number of regions, each region
should be no greater than 32K on the average ( = (2^20 / 2^17) * 4K).
Your regions are 100K on the average, so you are using 100K / 4K = 25 mtt entries
per region on the average (instead of 8).  

The calculation for 32K regions is (roughly):
2^15 regions * 2^5 mtt's per region = 2^20 mtts -- the max value.

- Jack

> Below is the output of lspci and ibv_devinfo -v. I also recently upgraded to
> the latest f/w but that didn't make a difference.
> 
> 
> lspci | grep Mellanox
> 01:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor
> compatibility mode) (rev 20)
> 
> ibv_devinfo -v
> hca_id: mthca0
>         fw_ver:                         4.8.200
>         node_guid:                      0006:6a00:9800:8403
>         sys_image_guid:                 0006:6a00:9800:8403
>         vendor_id:                      0x02c9
>         vendor_part_id:                 25208
>         hw_ver:                         0xA0
>         board_id:                       MT_0200000001
>         phys_port_cnt:                  2
>         max_mr_size:                    0xffffffffffffffff
>         page_size_cap:                  0xfffff000
>         max_qp:                         64512
>         max_qp_wr:                      65535
>         device_cap_flags:               0x00001c76
>         max_sge:                        59
>         max_sge_rd:                     0
>         max_cq:                         65408
>         max_cqe:                        131071
>         max_mr:                         131056
>         max_pd:                         32768
>         max_qp_rd_atom:                 4
>         max_ee_rd_atom:                 0
>         max_res_rd_atom:                258048
>         max_qp_init_rd_atom:            128
>         max_ee_init_rd_atom:            0
>         atomic_cap:                     ATOMIC_HCA (1)
>         max_ee:                         0
>         max_rdd:                        0
>         max_mw:                         0
>         max_raw_ipv6_qp:                0
>         max_raw_ethy_qp:                0
>         max_mcast_grp:                  8192
>         max_mcast_qp_attach:            56
>         max_total_mcast_qp_attach:      458752
>         max_ah:                         0
>         max_fmr:                        0
>         max_srq:                        960
>         max_srq_wr:                     65535
>         max_srq_sge:                    31
>         max_pkeys:                      64
>         local_ca_ack_delay:             15
> 
> 
> 
> On Tue, Mar 11, 2008 at 11:35 PM, Jack Morgenstein <jackm at dev.mellanox.co.il>
> wrote:
> 
> > On Tuesday 11 March 2008 19:38, Rajouri Jammu wrote:
> > > I think it's Arabel.
> > >
> > > Both drivers are loaded (ib_mthca and mlx4_core). How do I tell which
> > > driver's settings I should modify?
> > >
> > > What will be the max_mr value if log_num_mpt = 20?
> > >
> >
> > To see which device you have, type the following command in your linux
> > console:
> >
> > lspci | grep Mellanox
> >
> > For ConnectX (mlx4), you will see:
> > InfiniBand: Mellanox Technologies: Unknown device 634a (rev a0)
> >  (lspci has not caught up with us yet).
> >
> > For Arbel (InfiniHost III), you will see either:
> > InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex HCA (rev a0)
> >
> > or, if you are running your arbel in Tavor compatibility mode:
> >
> > InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor
> > compatibility mode) (rev 20)
> > ===========
> > If your installed HCA is a ConnectX, you should use the module parameters
> > for mlx4.
> > If your installed HCA is an InfiniHost III, you should use the module
> > parameters for ib_mthca.
> >
> > - Jack
> >
>