[ofa-general] RNIC resource limits
Steve Wise
swise at opengridcomputing.com
Wed Jun 4 07:22:48 PDT 2008
> [root at achilles ~]# ibv_devinfo -v
> hca_id: cxgb3_0
> fw_ver: 0.0.0
> node_guid: 0007:4301:33f7:0000
> sys_image_guid: 0007:4301:33f7:0000
> vendor_id: 0x1425
> vendor_part_id: 49
> hw_ver: 0x0
> board_id: 1425.31
> phys_port_cnt: 2
> max_mr_size: 0xffffffffffffffff
> page_size_cap: 0x0
> max_qp: 32736
> max_qp_wr: 16777215
> device_cap_flags: 0x00038000
> max_sge: 4
> max_sge_rd: 1
> max_cq: 32767
> max_cqe: 16777215
> max_mr: 32768
> max_pd: 32767
> max_qp_rd_atom: 8
> max_ee_rd_atom: 0
> max_res_rd_atom: 0
> max_qp_init_rd_atom: 8
> max_ee_init_rd_atom: 0
> atomic_cap: ATOMIC_NONE (0)
> max_ee: 0
> max_rdd: 0
> max_mw: 0
> max_raw_ipv6_qp: 0
> max_raw_ethy_qp: 0
> max_mcast_grp: 0
> max_mcast_qp_attach: 0
> max_total_mcast_qp_attach: 0
> max_ah: 0
> max_fmr: 0
> max_srq: 0
> max_pkeys: 0
> local_ca_ack_delay: 0
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: invalid MTU (225)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> max_msg_sz: 0xffffffff
> port_cap_flags: 0x009f0000
> max_vl_num: invalid value (255)
> bad_pkey_cntr: 0x213
> qkey_viol_cntr: 0x0
> sm_sl: 0
> pkey_tbl_len: 1
> gid_tbl_len: 1
> subnet_timeout: 146
> init_type_reply: 39
> active_width: 4X (2)
> active_speed: 5.0 Gbps (2)
> phys_state: invalid physical state (0)
>
> port: 2
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: invalid MTU (225)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> max_msg_sz: 0xffffffff
> port_cap_flags: 0x009f0000
> max_vl_num: invalid value (255)
> bad_pkey_cntr: 0x213
> qkey_viol_cntr: 0x0
> sm_sl: 0
> pkey_tbl_len: 1
> gid_tbl_len: 1
> subnet_timeout: 146
> init_type_reply: 39
> active_width: 4X (2)
> active_speed: 5.0 Gbps (2)
> phys_state: invalid physical state (0)
>
> When creating a QP, I need to specify some capacity information:
> struct ibv_qp_cap {
> uint32_t max_send_wr;
> uint32_t max_recv_wr;
> uint32_t max_send_sge;
> uint32_t max_recv_sge;
> uint32_t max_inline_data;
> };
>
> According to the above listing, I should be able to use:
> 16777215 WRs (max_qp_wr: 16777215) [is this per qp
> or total?]
> 4 SGEs (max_sge: 4)
>
> It does not say anything about the max_inline_data.
>
max inline supported is 64.
> I have tried to create a QP with as many resources as possible and found
> the following:
> max_send_wr cannot exceed 16384
> max_recv_wr cannot exceed 1023
> max_send_sge cannot exceed 4294967295 (maximum a uint32_t can hold)
> max_recv_sge cannot exceed 4294967295 (maximum a uint32_t can hold)
> max_inline_data cannot exceed 64
>
The limits described in the device attributes are max supported by the
HW. The OS limits these to much lower. For example, the chelsio work
queue must be in contiguous dma-coherent memory. Linux pretty much
limits that to a max size of about 128KB. That boils down to the
max_send_wr and max_recv_wr limits you are experiencing.
Note that the chelsio device uses a single work queue to implement both
the SQ and RQ. So if you set your max_send_wr to say 16, then you will
be able to set max_recv_wr much larger than 1023. So the two max depths
inter-relate, and are combined to allocate a single work queue. Hope
that makes sense.
send_sge and recv_sge are 4 max. the chelsio driver should be failing
the create_qp with the values you are passing in. That's a bug.
> If the stated limits are exceeded, the call to ibv_qp_create() fails.
>
> I am wondering now, if I can really use as many WRs, SGEs and inline
> data as the figures
> above or not. It is also not clear to me if these figures represent
> per-QP values or
> if they are global max values across all QPs.
>
You should use 4 for SGE depths and I would keep the SQ and RQ as
shallow as possible.
Steve.
More information about the general
mailing list