<br><tt><font size=2>Hello,</font></tt>
<br>
<br><tt><font size=2>I have asked this question about RNIC resource limits
before:</font></tt>
<br>
<br><tt><font size=2><snip></font></tt>
<br><tt><font size=2><br>
> > Could you give me some insight in what the limits of the Chelsio
RNIC <br>
> > are? (Max MRs, QPs, PDs etc)<br>
> ><br>
> > Many thanks and kind regards,<br>
> > Philip <br>
></font></tt>
<br>
<br><tt><font size=2><snip></font></tt>
<br><tt><font size=2><br>
> Try running ibv_devinfo -v to see driver/hw limits.<br>
><br>
> However, how are you limited? Are you getting failures registering
<br>
> memory? Did you try setting your ulimit -l to unlimited or at
least as <br>
> large as the memory region you want to register?<br>
><br>
> Steve.<br>
</font></tt>
<br><tt><font size=2>Steve, thanks for the answer!<br>
</font></tt>
<br><tt><font size=2>When running 'ibv_devinfo -v' on my Chelsio RNIC (T3)
with OFED 1.3 (FW 5.0)</font></tt>
<br><tt><font size=2>I get the following:</font></tt>
<br>
<br><tt><font size=2>[root@achilles ~]# ibv_devinfo -v</font></tt>
<br><tt><font size=2>hca_id: cxgb3_0</font></tt>
<br><tt><font size=2> fw_ver:
0.0.0</font></tt>
<br><tt><font size=2> node_guid:
0007:4301:33f7:0000</font></tt>
<br><tt><font size=2> sys_image_guid:
0007:4301:33f7:0000</font></tt>
<br><tt><font size=2> vendor_id:
0x1425</font></tt>
<br><tt><font size=2> vendor_part_id:
49</font></tt>
<br><tt><font size=2> hw_ver:
0x0</font></tt>
<br><tt><font size=2> board_id:
1425.31</font></tt>
<br><tt><font size=2> phys_port_cnt:
2</font></tt>
<br><tt><font size=2> max_mr_size:
0xffffffffffffffff</font></tt>
<br><tt><font size=2> page_size_cap:
0x0</font></tt>
<br><tt><font size=2> max_qp:
32736</font></tt>
<br><tt><font size=2> max_qp_wr:
16777215</font></tt>
<br><tt><font size=2> device_cap_flags:
0x00038000</font></tt>
<br><tt><font size=2> max_sge:
4</font></tt>
<br><tt><font size=2> max_sge_rd:
1</font></tt>
<br><tt><font size=2> max_cq:
32767</font></tt>
<br><tt><font size=2> max_cqe:
16777215</font></tt>
<br><tt><font size=2> max_mr:
32768</font></tt>
<br><tt><font size=2> max_pd:
32767</font></tt>
<br><tt><font size=2> max_qp_rd_atom:
8</font></tt>
<br><tt><font size=2> max_ee_rd_atom:
0</font></tt>
<br><tt><font size=2> max_res_rd_atom:
0</font></tt>
<br><tt><font size=2> max_qp_init_rd_atom:
8</font></tt>
<br><tt><font size=2> max_ee_init_rd_atom:
0</font></tt>
<br><tt><font size=2> atomic_cap:
ATOMIC_NONE (0)</font></tt>
<br><tt><font size=2> max_ee:
0</font></tt>
<br><tt><font size=2> max_rdd:
0</font></tt>
<br><tt><font size=2> max_mw:
0</font></tt>
<br><tt><font size=2> max_raw_ipv6_qp:
0</font></tt>
<br><tt><font size=2> max_raw_ethy_qp:
0</font></tt>
<br><tt><font size=2> max_mcast_grp:
0</font></tt>
<br><tt><font size=2> max_mcast_qp_attach:
0</font></tt>
<br><tt><font size=2> max_total_mcast_qp_attach:
0</font></tt>
<br><tt><font size=2> max_ah:
0</font></tt>
<br><tt><font size=2> max_fmr:
0</font></tt>
<br><tt><font size=2> max_srq:
0</font></tt>
<br><tt><font size=2> max_pkeys:
0</font></tt>
<br><tt><font size=2> local_ca_ack_delay:
0</font></tt>
<br><tt><font size=2>
port: 1</font></tt>
<br><tt><font size=2>
state:
PORT_ACTIVE (4)</font></tt>
<br><tt><font size=2>
max_mtu:
4096 (5)</font></tt>
<br><tt><font size=2>
active_mtu:
invalid MTU (225)</font></tt>
<br><tt><font size=2>
sm_lid:
0</font></tt>
<br><tt><font size=2>
port_lid:
0</font></tt>
<br><tt><font size=2>
port_lmc:
0x00</font></tt>
<br><tt><font size=2>
max_msg_sz:
0xffffffff</font></tt>
<br><tt><font size=2>
port_cap_flags:
0x009f0000</font></tt>
<br><tt><font size=2>
max_vl_num:
invalid value (255)</font></tt>
<br><tt><font size=2>
bad_pkey_cntr:
0x213</font></tt>
<br><tt><font size=2>
qkey_viol_cntr:
0x0</font></tt>
<br><tt><font size=2>
sm_sl:
0</font></tt>
<br><tt><font size=2>
pkey_tbl_len:
1</font></tt>
<br><tt><font size=2>
gid_tbl_len:
1</font></tt>
<br><tt><font size=2>
subnet_timeout:
146</font></tt>
<br><tt><font size=2>
init_type_reply: 39</font></tt>
<br><tt><font size=2>
active_width:
4X (2)</font></tt>
<br><tt><font size=2>
active_speed:
5.0 Gbps (2)</font></tt>
<br><tt><font size=2>
phys_state:
invalid physical state (0)</font></tt>
<br>
<br><tt><font size=2>
port: 2</font></tt>
<br><tt><font size=2>
state:
PORT_ACTIVE (4)</font></tt>
<br><tt><font size=2>
max_mtu:
4096 (5)</font></tt>
<br><tt><font size=2>
active_mtu:
invalid MTU (225)</font></tt>
<br><tt><font size=2>
sm_lid:
0</font></tt>
<br><tt><font size=2>
port_lid:
0</font></tt>
<br><tt><font size=2>
port_lmc:
0x00</font></tt>
<br><tt><font size=2>
max_msg_sz:
0xffffffff</font></tt>
<br><tt><font size=2>
port_cap_flags:
0x009f0000</font></tt>
<br><tt><font size=2>
max_vl_num:
invalid value (255)</font></tt>
<br><tt><font size=2>
bad_pkey_cntr:
0x213</font></tt>
<br><tt><font size=2>
qkey_viol_cntr:
0x0</font></tt>
<br><tt><font size=2>
sm_sl:
0</font></tt>
<br><tt><font size=2>
pkey_tbl_len:
1</font></tt>
<br><tt><font size=2>
gid_tbl_len:
1</font></tt>
<br><tt><font size=2>
subnet_timeout:
146</font></tt>
<br><tt><font size=2>
init_type_reply: 39</font></tt>
<br><tt><font size=2>
active_width:
4X (2)</font></tt>
<br><tt><font size=2>
active_speed:
5.0 Gbps (2)</font></tt>
<br><tt><font size=2>
phys_state:
invalid physical state (0)</font></tt>
<br>
<br><tt><font size=2>When creating a QP, I need to specify some capacity
information:</font></tt>
<br><tt><font size=2>struct ibv_qp_cap {</font></tt>
<br><tt><font size=2> uint32_t
max_send_wr;</font></tt>
<br><tt><font size=2> uint32_t
max_recv_wr;</font></tt>
<br><tt><font size=2> uint32_t
max_send_sge;</font></tt>
<br><tt><font size=2> uint32_t
max_recv_sge;</font></tt>
<br><tt><font size=2> uint32_t
max_inline_data;</font></tt>
<br><tt><font size=2>};</font></tt>
<br>
<br><tt><font size=2>According to the above listing, I should be able to
use:</font></tt>
<br><tt><font size=2>16777215 WRs
(max_qp_wr: 16777215)
[is this per qp or total?]</font></tt>
<br><tt><font size=2>4
SGEs (max_sge:
4)</font></tt>
<br>
<br><tt><font size=2>It does not say anything about the max_inline_data.</font></tt>
<br>
<br><tt><font size=2>I have tried to create a QP with as many resources
as possible and found the following:</font></tt>
<br><tt><font size=2>max_send_wr
cannot exceed 16384</font></tt>
<br><tt><font size=2>max_recv_wr
cannot exceed 1023</font></tt>
<br><tt><font size=2>max_send_sge cannot
exceed 4294967295 (maximum a uint32_t can hold)</font></tt>
<br><tt><font size=2>max_recv_sge cannot
exceed 4294967295 (maximum a uint32_t can hold)</font></tt>
<br><tt><font size=2>max_inline_data cannot
exceed 64</font></tt>
<br>
<br><tt><font size=2>If the stated limits are exceeded, the call to ibv_qp_create()
fails.</font></tt>
<br>
<br><tt><font size=2>I am wondering now, if I can really use as many WRs,
SGEs and inline data as the figures</font></tt>
<br><tt><font size=2>above or not. It is also not clear to me if these
figures represent per-QP values or</font></tt>
<br><tt><font size=2>if they are global max values across all QPs.</font></tt>
<br>
<br><tt><font size=2>Many thanks for you advice and best regards,</font></tt>
<br><tt><font size=2> Philip</font></tt>