<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">We updated to MLNX_OFED_LINUX-1.5.3-3.0.0 from OFED-1.5.2_1 recently and we're running into issues where we get a ENOMEM error from ibv_create_qp(). Earlier with this OFED version we were running into a ENOMEM with ibv_reg_mr() but that was resolved by setting log_num_mtt to 24 as described in this link http://h10025.www1.hp.com/ewfrf/wc/document?cc=us&lc=en&dlc=en&docname=c03113904</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">All the other settings in /sys/module/mlx4_core/parameters are the default values. The node itself has 96GB of memory we're nowhere close
to using that. Is there any way to figure out whats going on ?</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">thanks</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">K. Can</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">here's the output from ibv_devinfo -v:</div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div><div><font face="'times new roman', 'new york', times, serif">hca_id: mlx4_0</font></div><div><font face="'times new roman', 'new york', times, serif"> transport:
InfiniBand (0)</font></div><div><font face="'times new roman', 'new york', times, serif"> fw_ver: 2.9.1000</font></div><div><font face="'times new roman', 'new york', times, serif"> node_guid: 0002:c903:004b:d5e6</font></div><div><font face="'times new roman', 'new york', times, serif"> sys_image_guid: 0002:c903:004b:d5e9</font></div><div><font face="'times new roman', 'new york', times, serif"> vendor_id: 0x02c9</font></div><div><font face="'times new roman', 'new york', times, serif"> vendor_part_id:
26428</font></div><div><font face="'times new roman', 'new york', times, serif"> hw_ver: 0xB0</font></div><div><font face="'times new roman', 'new york', times, serif"> board_id: HP_0180000009</font></div><div><font face="'times new roman', 'new york', times, serif"> phys_port_cnt: 2</font></div><div><font face="'times new roman', 'new york', times, serif"> max_mr_size: 0xffffffffffffffff</font></div><div><font face="'times new roman', 'new york', times, serif"> page_size_cap:
0xfffffe00</font></div><div><font face="'times new roman', 'new york', times, serif"> max_qp: 260032</font></div><div><font face="'times new roman', 'new york', times, serif"> max_qp_wr: 16351</font></div><div><font face="'times new roman', 'new york', times, serif"> device_cap_flags: 0x007c9c76</font></div><div><font face="'times new roman', 'new york', times, serif"> max_sge: 32</font></div><div><font face="'times new roman', 'new york', times, serif"> max_sge_rd:
0</font></div><div><font face="'times new roman', 'new york', times, serif"> max_cq: 65408</font></div><div><font face="'times new roman', 'new york', times, serif"> max_cqe: 4194303</font></div><div><font face="'times new roman', 'new york', times, serif"> max_mr: 524272</font></div><div><font face="'times new roman', 'new york', times, serif"> max_pd: 32764</font></div><div><font face="'times new roman', 'new york', times, serif"> max_qp_rd_atom:
16</font></div><div><font face="'times new roman', 'new york', times, serif"> max_ee_rd_atom: 0</font></div><div><font face="'times new roman', 'new york', times, serif"> max_res_rd_atom: 4160512</font></div><div><font face="'times new roman', 'new york', times, serif"> max_qp_init_rd_atom: 128</font></div><div><font face="'times new roman', 'new york', times, serif"> max_ee_init_rd_atom: 0</font></div><div><font face="'times new roman', 'new york', times, serif"> atomic_cap: ATOMIC_HCA
(1)</font></div><div><font face="'times new roman', 'new york', times, serif"> max_ee: 0</font></div><div><font face="'times new roman', 'new york', times, serif"> max_rdd: 0</font></div><div><font face="'times new roman', 'new york', times, serif"> max_mw: 0</font></div><div><font face="'times new roman', 'new york', times, serif"> max_raw_ipv6_qp: 0</font></div><div><font face="'times new roman', 'new york', times, serif"> max_raw_ethy_qp: 2</font></div><div><font
face="'times new roman', 'new york', times, serif"> max_mcast_grp: 8192</font></div><div><font face="'times new roman', 'new york', times, serif"> max_mcast_qp_attach: 120</font></div><div><font face="'times new roman', 'new york', times, serif"> max_total_mcast_qp_attach: 983040</font></div><div><font face="'times new roman', 'new york', times, serif"> max_ah: 0</font></div><div><font face="'times new roman', 'new york', times, serif"> max_fmr: 0</font></div><div><font face="'times new roman', 'new york', times, serif">
max_srq: 65472</font></div><div><font face="'times new roman', 'new york', times, serif"> max_srq_wr: 16383</font></div><div><font face="'times new roman', 'new york', times, serif"> max_srq_sge: 31</font></div><div><font face="'times new roman', 'new york', times, serif"> max_pkeys: 128</font></div><div><font face="'times new roman', 'new york', times, serif"> local_ca_ack_delay: 15</font></div><div><font face="'times new roman', 'new york', times, serif">
port: 1</font></div><div><font face="'times new roman', 'new york', times, serif"> state: PORT_ACTIVE (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> max_mtu: 2048 (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> active_mtu: 2048 (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> sm_lid:
9</font></div><div><font face="'times new roman', 'new york', times, serif"> port_lid: 311</font></div><div><font face="'times new roman', 'new york', times, serif"> port_lmc: 0x00</font></div><div><font face="'times new roman', 'new york', times, serif"> link_layer: IB</font></div><div><font face="'times new roman', 'new york', times, serif"> max_msg_sz: 0x40000000</font></div><div><font face="'times new roman', 'new york', times, serif">
port_cap_flags: 0x02510868</font></div><div><font face="'times new roman', 'new york', times, serif"> max_vl_num: 8 (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> bad_pkey_cntr: 0x0</font></div><div><font face="'times new roman', 'new york', times, serif"> qkey_viol_cntr: 0x0</font></div><div><font face="'times new roman', 'new york', times, serif"> sm_sl:
0</font></div><div><font face="'times new roman', 'new york', times, serif"> pkey_tbl_len: 128</font></div><div><font face="'times new roman', 'new york', times, serif"> gid_tbl_len: 128</font></div><div><font face="'times new roman', 'new york', times, serif"> subnet_timeout: 18</font></div><div><font face="'times new roman', 'new york', times, serif"> init_type_reply: 0</font></div><div><font face="'times new roman', 'new york', times, serif">
active_width: 4X (2)</font></div><div><font face="'times new roman', 'new york', times, serif"> active_speed: 10.0 Gbps (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> phys_state: LINK_UP (5)</font></div><div><font face="'times new roman', 'new york', times, serif"> GID[ 0]: fe80:0000:0000:0000:0002:c903:004b:d5e7</font></div><div><font face="'times new roman', 'new york', times, serif"><br></font></div><div><font face="'times new roman', 'new york', times,
serif"> port: 2</font></div><div><font face="'times new roman', 'new york', times, serif"> state: PORT_ACTIVE (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> max_mtu: 2048 (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> active_mtu: 2048 (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> sm_lid:
9</font></div><div><font face="'times new roman', 'new york', times, serif"> port_lid: 158</font></div><div><font face="'times new roman', 'new york', times, serif"> port_lmc: 0x00</font></div><div><font face="'times new roman', 'new york', times, serif"> link_layer: IB</font></div><div><font face="'times new roman', 'new york', times, serif"> max_msg_sz: 0x40000000</font></div><div><font face="'times new roman', 'new
york', times, serif"> port_cap_flags: 0x02510868</font></div><div><font face="'times new roman', 'new york', times, serif"> max_vl_num: 8 (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> bad_pkey_cntr: 0x0</font></div><div><font face="'times new roman', 'new york', times, serif"> qkey_viol_cntr: 0x0</font></div><div><font face="'times new roman', 'new york', times, serif"> sm_sl:
0</font></div><div><font face="'times new roman', 'new york', times, serif"> pkey_tbl_len: 128</font></div><div><font face="'times new roman', 'new york', times, serif"> gid_tbl_len: 128</font></div><div><font face="'times new roman', 'new york', times, serif"> subnet_timeout: 18</font></div><div><font face="'times new roman', 'new york', times, serif"> init_type_reply: 0</font></div><div><font face="'times new roman', 'new york', times,
serif"> active_width: 4X (2)</font></div><div><font face="'times new roman', 'new york', times, serif"> active_speed: 10.0 Gbps (4)</font></div><div><font face="'times new roman', 'new york', times, serif"> phys_state: LINK_UP (5)</font></div><div><font face="'times new roman', 'new york', times, serif"> GID[ 0]: fe80:0000:0000:0000:0002:c903:004b:d5e8</font></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;
"><br></div></div></div></body></html>