[ewg] ibv_create_qp fails with enomem

[Ker Can]

Bob, 

Our max locked memory was already set to unlimited. 

thanks
K. Can



----- Original Message -----
From: Robert D. Russell <rdr at iol.unh.edu>
To: Ker Can <kercan7478 at yahoo.com>
Cc: "ewg at lists.openfabrics.org" <ewg at lists.openfabrics.org>
Sent: Friday, March 23, 2012 5:08 PM
Subject: Re: [ewg] ibv_create_qp fails with enomem

Ker:

In your /etc/security/limits.conf file, try adding the lines
*  hard  memlock  unlimited
*  soft  memlock  unlimited

We hit the same problem and this cured it.

Bob Russell
rdr at iol.unh.edu


On Fri, 23 Mar 2012, Ker Can wrote:

> Sorry about all the formatting - trying again with plain text.
>
> We updated to MLNX_OFED_LINUX-1.5.3-3.0.0 from OFED-1.5.2_1  recently and we're running into issues where we get a ENOMEM error from ibv_create_qp(). Earlier with this OFED version we were running into a ENOMEM with ibv_reg_mr() but that was resolved by setting log_num_mtt to 24 as described in this link  
> http://h10025.www1.hp.com/ewfrf/wc/document?cc=us&lc=en&dlc=en&docname=c03113904
> All the other settings in /sys/module/mlx4_core/parameters are the default values.   The node itself has 96GB of memory we're nowhere close to using that. Is there any way to figure out whats going on ?
>
> thanks
> K. Can
>
> here's the output from ibv_devinfo -v:
>
> hca_id: mlx4_0
>         transport:                      InfiniBand (0)
>         fw_ver:                         2.9.1000
>         node_guid:                      0002:c903:004b:d5e6
>         sys_image_guid:                 0002:c903:004b:d5e9
>         vendor_id:                      0x02c9
>         vendor_part_id:                 26428
>         hw_ver:                         0xB0
>         board_id:                       HP_0180000009
>         phys_port_cnt:                  2
>         max_mr_size:                    0xffffffffffffffff
>         page_size_cap:                  0xfffffe00
>         max_qp:                         260032
>         max_qp_wr:                      16351
>         device_cap_flags:               0x007c9c76
>         max_sge:                        32
>         max_sge_rd:                     0
>         max_cq:                         65408
>         max_cqe:                        4194303
>         max_mr:                         524272
>         max_pd:                         32764
>         max_qp_rd_atom:                 16
>         max_ee_rd_atom:                 0
>         max_res_rd_atom:                4160512
>         max_qp_init_rd_atom:            128
>         max_ee_init_rd_atom:            0
>         atomic_cap:                     ATOMIC_HCA (1)
>         max_ee:                         0
>         max_rdd:                        0
>         max_mw:                         0
>         max_raw_ipv6_qp:                0
>         max_raw_ethy_qp:                2
>         max_mcast_grp:                  8192
>         max_mcast_qp_attach:            120
>         max_total_mcast_qp_attach:      983040
>         max_ah:                         0
>         max_fmr:                        0
>         max_srq:                        65472
>         max_srq_wr:                     16383
>         max_srq_sge:                    31
>         max_pkeys:                      128
>         local_ca_ack_delay:             15
>                 port:   1
>                         state:                  PORT_ACTIVE (4)
>                         max_mtu:                2048 (4)
>                         active_mtu:             2048 (4)
>                         sm_lid:                 9
>                         port_lid:               311
>                         port_lmc:               0x00
>                         link_layer:             IB
>                         max_msg_sz:             0x40000000
>                         port_cap_flags:         0x02510868
>                         max_vl_num:             8 (4)
>                         bad_pkey_cntr:          0x0
>                         qkey_viol_cntr:         0x0
>                         sm_sl:                  0
>                         pkey_tbl_len:           128
>                         gid_tbl_len:            128
>                         subnet_timeout:         18
>                         init_type_reply:        0
>                         active_width:           4X (2)
>                         active_speed:           10.0 Gbps (4)
>                         phys_state:             LINK_UP (5)
>                         GID[  0]:               fe80:0000:0000:0000:0002:c903:004b:d5e7
>
>                 port:   2
>                         state:                  PORT_ACTIVE (4)
>                         max_mtu:                2048 (4)
>                         active_mtu:             2048 (4)
>                         sm_lid:                 9
>                         port_lid:               158
>                         port_lmc:               0x00
>                         link_layer:             IB
>                         max_msg_sz:             0x40000000
>                         port_cap_flags:         0x02510868
>                         max_vl_num:             8 (4)
>                         bad_pkey_cntr:          0x0
>                         qkey_viol_cntr:         0x0
>                         sm_sl:                  0
>                         pkey_tbl_len:           128
>                         gid_tbl_len:            128
>                         subnet_timeout:         18
>                         init_type_reply:        0
>                         active_width:           4X (2)
>                         active_speed:           10.0 Gbps (4)
>                         phys_state:             LINK_UP (5)
>                         GID[  0]:               fe80:0000:0000:0000:0002:c903:004b:d5e8
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>