[ewg] ibv_create_qp fails with enomem

[Ira Weiny]

I hate to be this way but I feel I must mention this.

Mellanox OFED is their product.  If you are having trouble with their software you should contact them.

That said, why did you feel you needed to update to their version rather than the latest OFED or perhaps even using what is provided with a standard distro?

Ira


On Fri, 23 Mar 2012 13:47:46 -0700 (PDT)
Ker Can <kercan7478 at yahoo.com> wrote:

> Sorry about all the formatting - trying again with plain text.
> 
> We updated to MLNX_OFED_LINUX-1.5.3-3.0.0 from OFED-1.5.2_1  recently and we're running into issues where we get a ENOMEM error from ibv_create_qp(). Earlier with this OFED version we were running into a ENOMEM with ibv_reg_mr() but that was resolved by setting log_num_mtt to 24 as described in this link  
> http://h10025.www1.hp.com/ewfrf/wc/document?cc=us&lc=en&dlc=en&docname=c03113904
> All the other settings in /sys/module/mlx4_core/parameters are the default values.   The node itself has 96GB of memory we're nowhere close to using that. Is there any way to figure out whats going on ?
> 
> thanks
> K. Can
> 
> here's the output from ibv_devinfo -v:
> 
> hca_id: mlx4_0
>         transport:                      InfiniBand (0)
>         fw_ver:                         2.9.1000
>         node_guid:                      0002:c903:004b:d5e6
>         sys_image_guid:                 0002:c903:004b:d5e9
>         vendor_id:                      0x02c9
>         vendor_part_id:                 26428
>         hw_ver:                         0xB0
>         board_id:                       HP_0180000009
>         phys_port_cnt:                  2
>         max_mr_size:                    0xffffffffffffffff
>         page_size_cap:                  0xfffffe00
>         max_qp:                         260032
>         max_qp_wr:                      16351
>         device_cap_flags:               0x007c9c76
>         max_sge:                        32
>         max_sge_rd:                     0
>         max_cq:                         65408
>         max_cqe:                        4194303
>         max_mr:                         524272
>         max_pd:                         32764
>         max_qp_rd_atom:                 16
>         max_ee_rd_atom:                 0
>         max_res_rd_atom:                4160512
>         max_qp_init_rd_atom:            128
>         max_ee_init_rd_atom:            0
>         atomic_cap:                     ATOMIC_HCA (1)
>         max_ee:                         0
>         max_rdd:                        0
>         max_mw:                         0
>         max_raw_ipv6_qp:                0
>         max_raw_ethy_qp:                2
>         max_mcast_grp:                  8192
>         max_mcast_qp_attach:            120
>         max_total_mcast_qp_attach:      983040
>         max_ah:                         0
>         max_fmr:                        0
>         max_srq:                        65472
>         max_srq_wr:                     16383
>         max_srq_sge:                    31
>         max_pkeys:                      128
>         local_ca_ack_delay:             15
>                 port:   1
>                         state:                  PORT_ACTIVE (4)
>                         max_mtu:                2048 (4)
>                         active_mtu:             2048 (4)
>                         sm_lid:                 9
>                         port_lid:               311
>                         port_lmc:               0x00
>                         link_layer:             IB
>                         max_msg_sz:             0x40000000
>                         port_cap_flags:         0x02510868
>                         max_vl_num:             8 (4)
>                         bad_pkey_cntr:          0x0
>                         qkey_viol_cntr:         0x0
>                         sm_sl:                  0
>                         pkey_tbl_len:           128
>                         gid_tbl_len:            128
>                         subnet_timeout:         18
>                         init_type_reply:        0
>                         active_width:           4X (2)
>                         active_speed:           10.0 Gbps (4)
>                         phys_state:             LINK_UP (5)
>                         GID[  0]:               fe80:0000:0000:0000:0002:c903:004b:d5e7
> 
>                 port:   2
>                         state:                  PORT_ACTIVE (4)
>                         max_mtu:                2048 (4)
>                         active_mtu:             2048 (4)
>                         sm_lid:                 9
>                         port_lid:               158
>                         port_lmc:               0x00
>                         link_layer:             IB
>                         max_msg_sz:             0x40000000
>                         port_cap_flags:         0x02510868
>                         max_vl_num:             8 (4)
>                         bad_pkey_cntr:          0x0
>                         qkey_viol_cntr:         0x0
>                         sm_sl:                  0
>                         pkey_tbl_len:           128
>                         gid_tbl_len:            128
>                         subnet_timeout:         18
>                         init_type_reply:        0
>                         active_width:           4X (2)
>                         active_speed:           10.0 Gbps (4)
>                         phys_state:             LINK_UP (5)
>                         GID[  0]:               fe80:0000:0000:0000:0002:c903:004b:d5e8
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg


-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2 at llnl.gov